Bortlog


Channel's geo and language: Russia, Russian
Category: Blogs


Captain: @bstorozhuk
Ви можете читати цей блог українською: https://t.me/bortlogua

Related channels

Channel's geo and language
Russia, Russian
Category
Blogs
Statistics
Posts filter


Forward from: FEDOROV
Помощь нашим бойцам в крипте Ethereum. Bitcoin and Tether (USDTtrc20):

BTC - 357a3So9CbsNfBBgFYACGvxxS6tMaDoa1P

ETH - 0x165CD37b4C644C2921454429E7F9358d18A45e14

USDT (trc20) - TEFccmfQ38cZS1DTZVhsxKVDckA8Y6VfCy


The content is hidden


Dear friends,

Russia has invaded Ukraine and started a full-scale European war. As you reading this message - Ukrainian cities are being bombarded by Russian missiles, menacing troops are marching over the borders and arriving via the Black Sea as citizens are desperately trying to flee towards the west. 

Ukraine, like never before needs your help! 

We hope that world leaders will unite in their response to stop Russia and help Ukraine. 

We believe in the combined power of regular people like you and me. Every one of us can help Ukraine now. 

Here’s how we can all help…

1. Send a donation. Every single donation, no matter how small, helps:
- Donation account of the National Bank of Ukraine https://bank.gov.ua/en/news/all/natsionalniy-bank-vidkriv-spetsrahunok-dlya-zboru-koshtiv-na-potrebi-armiyi
 - Charity that supports war victims and veterans https://savelife.in.ua/en/donate/  

2. Ask your elected representatives to support Ukraine and impose stronger sanction against Russia.

3. Support the people of Ukraine by joining local demonstrations against Russian aggression.

4. Stop Russian disinformation, don't let them get away with it this time. Use "Report" functionality on social networks to block Russian bots. Contact your internet providers to block Russian sites with misinformation.

5. Share this message with your friends and ask them to spread it further.

We are stronger when united! Together we can win!




Forward from: CatOps
CatOps exists for almost five years already as a non-profit hobby project. Yes, at some point we were advertising some technical events here, but we never took money for that.

In my opinion, keeping it independent of ad money is what makes all the fun. I'll be honest with you, from time to time I get ideas of starting a Patreon or something similar, but each time it turns out to be too complicated.

However, there is a way to support CatOps financially! Even though this is an indirect support!

Consider subscribing to the Patreon page of the "Come Back Alive" Foundation or you can, of course, donate directly. They have all the details on their website. Unfortunately, not all the information on the website is translated to English, but the main things are there.

This Foundation supports Ukrainian army as well as establish re-integration programs for the veterans.

By supporting people, who take care of us, you empower us in doing our day-to-day jobs and providing some platform engineering content here.

With love 💛💙
@grem1in


This is a rare good example of rewriting something in Rust that seems to be justifiable https://pngquant.org/rust.html


Here is an article on how to track and get health metrics from critical infrastructure repositories: https://estebank.github.io/sustainable-growth-and-visibility.html
The author suggests tracking different things like issue backlog, changes in the number of contributors, frequency, and speed of code reviews. From that data, you can derive certain knowledge about potential burnout of core contributors, or find some new projects that grow in popularity and become critical, so now require more contributors and attention.
This article is focused on Rust ecosystems, but the approach seems to be useful and universal for other ecosystems too.

It reminded me how Rob Pike a few years ago suggested how we can calculate criticality scores for open source projects and packages: https://github.com/ossf/criticality_score/blob/main/Quantifying_criticality_algorithm.pdf
Here you can also find and download pre-aggregated lists for different languages and ecosystems: https://github.com/ossf/criticality_score

If we combine these two ideas, it should be possible to proactively discover and offer some help to projects that need more hands or other resources.


Recently I've got a few interns to look after and mentor, so one of them asked for some advice on improving their system design skills.
So I gave a long monolog on digging through the layers of abstractions and being curious and genuinely understanding how things work. After it, I suggested them to read a "Designing Data-Intensive Applications" book and also prepared a shortlist of talks that can work as conversation starter or places from where you can start digging deeper.
Here is the list:

https://youtu.be/WE9c9AZe-DY
https://youtu.be/j6ow-UemzBc
https://youtu.be/1TIzPL4878Q
https://youtu.be/CZ3wIuvmHeM
https://youtu.be/I32hmY4diFY
https://youtu.be/hnpzNAPiC0E
https://youtu.be/y2j_TB3NsRc
https://youtu.be/t7iVCIYQbgk
https://youtu.be/IO4teCbHvZw
https://youtu.be/6w6E_B55p0E
https://youtu.be/B5NULPSiOGw
https://youtu.be/_M-oHxknfnI
https://youtu.be/lKXe3HUG2l4
https://youtu.be/r-TLSBdHe1A
https://youtu.be/fU9hR3kiOK0
https://youtu.be/lJ8ydIuPFeU
https://youtu.be/5ZjhNTM8XU8
https://youtu.be/9RMOc0SwRro
https://youtu.be/tM4YskS94b0
https://youtu.be/H0i_bXKwujQ
https://youtu.be/m64SWl9bfvk
https://youtu.be/cNICGEwmXLU
https://youtu.be/em9zLzM8O7c
https://youtu.be/ohvPnJYUW1E
https://youtu.be/HJIjTTHWYnE

Write in the comments: what else would you recommend or what talk you'd add to the list.


I have a great article for you about the challenges and problems of writing software for real-time audio generation.
https://glowcoil.com/posts/basedrop/

That's where we cannot do without low-level programming languages ​​with absolute control over what exactly will happen at any moment. Because you can't allow any unpredictable (non-constant) operations in the thread that generates the audio, it turns out that you can't do I/O, allocate or free memory, use mutexes or blocking queues to synchronize threads, etc.
With such restrictions, even the use of Rust looks too high-level because its RAII can play a bad joke on you and start to free some buffers when they lose their owner. So it turns out that even with Rust, you may, in special cases, have a memory leak or OOM.

It seems that for such software Zig with its explicit use of allocators and explicit defer, would be a little more convenient, but also not without difficulties.


Great article that shows what will happen if you do not design asynchronous programming features in low-level language from scratch https://tomaka.medium.com/a-look-back-at-asynchronous-rust-d54d63934a1c

It turns out that everything is kind of works, but there are a handful of annoying nuances that you have to put up with or fix with some dirty hacks. And it is similar in most languages and most runtimes. It seems to me that the industry is trying to solve the problem of expensive OS threads and context switches, not where it needs to be solved. Why can't we make OS threads more lightweight? That would be great, and we already know how to use treads in our streams and work with queues, semaphores, and mutexes. All our tooling content understands streams, then what's the problem?

In 2013, people from Google delivered this talk https://www.youtube.com/watch?v=KXuZi9aeGTw
Where they demonstrate how to reduce the overhead of context switch 100 times or more.
If I understand correctly, in 2013, they already used this solution for services written in C ++ using "regular" threads.

In 2020, they even offered 3 patches for Linux, but didn't finish the discussion and did not merge these patches, and it may take another 5-10 years to get back to this idea.


Well, if we are talking about all sorts of dynamic analysis to detect memory safety issues, then here is the news that I could not miss:
Linus merged to 5.12 the following mechanism https://lwn.net/Articles/835542/
And here are some explanations of how it works https://github.com/google/kasan/blob/master/KFENCE.md
In general, KFENCE is a kernel memory safety error detector with such a low overhead that it can be used in prod.


We had a really good conversation with @insert_reference_here in the comments for "Memory tagging" post here https://t.me/bortlog/81
And I think the arguments presented there are really compelling, but to some degree, they are based on the assumption that memory safety features based on type system and static analysis are essentially free.

But there are actual proofs that this is not true, and here is one of them:
As you may know, Microsoft is working on a C language extension that will bring memory safety features to it https://t.me/bortlog/23
They added 3 new pointer types, with a whole bunch on statical analysis, and at some points where it is not possible to prove safety staticaly, they add runtime null and bounds checks.

Microsoft Research team recently presented their results https://www.youtube.com/watch?v=EuxAzvtX9CI
and they showed the effects of the conversion of some C codebases to Checked C. On average, they change 18% of code during conversion, code size increases by 7%, and runtime overhead is almost 9%, and compile-time increased by 25%.

Of course, we can not draw any conclusions for only one research team work, but I'd say that it is safe to admit that in real-world examples, even type system and static analysis assisted memory safety is not free, and hardware support of some runtime checks with only


In case if you missed yesterday's announcement: Rustaceans are actively working to bring garbage collection to Rust.
Proof


Many people post a recent announcement of the ARMv9 architecture, but the most remarkable feature (in my opinion) has been announced for a long time now, and its support is even ready in compilers and operating systems. "Memory tagging" can be a revolutionary leap that will ensure the security and correctness of programs written in languages ​​with manual memory management.

For less than 5% CPU memory bandwidth, you get a 93% chance of intercepting global-buffer-overflow, heap-buffer-overflow, use-after-free, use-after-return, use-after-scope, stack-buffer-overflow. If your program works long enough and you have really good test coverage, you will catch all these bugs without any problems at all. In debug mode, you can pay 20% of bandwidth and get a full stack trace of where exactly, who and how violated the integrity, and so on.

In such a world, the benefits of big runtimes, overcomplicated programming languages, etc., partially disappear, and we again, solve everything on the silicon level))

In short, I highly recommend this talk; there are more details on how exactly this is implemented at the LLVM compiler and memory allocator in Android.

https://youtu.be/tqG9pr1K5G8


Here is a 2 years old article, which explains why the architecture of all modern operating systems is obsolete. The advent of new hardware and very high-speed network and disk interfaces, makes the strategy of running the kernel on all CPU cores and work with interfaces via system calls, disadvantageous due to the large number of context switches and the presence of shared memory.
To write really high-performant software, people are now doing kernel bypass, using eBPF or XDP.
And this is just one of the 5 points of criticism listed by the article authors.

The authors propose a new operating system architecture which they call parakernel. And this architecture will be as complementary as possible to all sorts of thread-per-core frameworks such as Seastar (http://seastar.io). Parakernel allocates all the necessary DRAM resources, network RX / TX queues, and disk NVMe queues when starting or initializing the process and guarantees their isolation and unique ownership by the process. It is proposed to implement network protocols in user space. Discard almost all POSIX APIs, especially various blocking operations, get rid of virtual address space, or leave it as a security feature, but ensure that the OS will make page replacements, swap memory pages, etc.
It turns out that the in such architecture kernel will only deal with resource allocation, handle process failures, and work as an adapter in case we run on some legacy hardware that does not support resource allocation at the driver or hardware level.

In short, it is a really cool article, and if you are interested in details, you can come here: https://penberg.org/parakernel-hotos19.pdf


Just a couple of hours ago, Oleg and I recorded a REST vs gRPC video.
As I promised, I am posting an extension video with a description of my thoughts on JSON indexing.
Do not take it too seriously, I just described some ideas, that haven't been verified in any way.
I did a little introduction there, but I suggest you watch the main podcast first.

https://youtu.be/jILVrU1w_yc


If you want to relax and watch how someone is codding, take a look at this channel https://youtube.com/c/AndreasKling
This person has been writing his operating system for a couple of years, but the last video is quite simple to watch https://youtu.be/X6s3XBWe2XU

To understand how TLB works and how exactly CPU and kernel agree on which pages are available for writing, or read-only, or execution, I advise https://open.spotify.com/episode/2jF3PrFnnyaeRYfteLTYeW?si=DyU6iHJDQ7aiYQ8rfq2jrA

Maybe Andreas himself has a video on the channel where he impements it 🤷🏻‍♂️


The new "This week in Rust" has an interesting link https://www.reddit.com/r/rust/comments/lg0a7b/benchmarking_tokio_tasks_and_goroutines/ to a discussion of how Go runtime very confidently wins over Tokio's when working with blocking system calls.
The top comment on Reddit explains why it is like that, and how to tweak Tokio's settings to almost eliminate the difference, and why these settings are not applied by default.
That is performance domination we deserve 😄


Michael Knyszek has rolled out a new GC pacer proposal for Go https://github.com/golang/go/issues/44167

Nothing magical is offered there, but the new algorithm will allow:
- Consider the number of goroutine stacks and global variables when choosing when to start Mark Phase.
- It will be possible to make a heap even less than 4MB.
- Well, there will be much fewer cases when the heap outgrows the planned limit.

Note: the detailed proposal contains interesting graphs with simulations of the behavior of the current and new pacer.


An interesting article (https://blog.cloudflare.com/why-we-started-putting-unpopular-assets-in-memory/) about how the team from Cloudflare divided their hybrid (disk + RAM) cache into separate generations to get rid of cases when a resource is requested once, gets into the cache, and then no one touches it there.
They have a very interesting idea, and a side effect in the form of a 20-25% reduction of disk writes.

But this is all done by the engineering specialists in the field)))
Meantime, researchers one year ago created a neural network for such hybrid caches (https://www.youtube.com/watch?v=WphwICSQPaQ&feature=youtu.be), which guesses what to add and where by keys.
And researchers managed to downgrade the records 6-25 times, depending on the load profile.

20 last posts shown.

291

subscribers
Channel statistics