This transcript - is mostly as is - edited lightly from the discussion on hiring for Rust which was held at the conference on 2nd November 2022. As per Chatham House Rules, names have not been mentioned in this transcript to prevent attribution of quotes to persons.

Cryptography companies prefer to hire developers good with cryptography and then train them on Rust language.

Emcees (E): What are the problems your face while hiring Rust programmers?

Audience (A): The obvious problem that everyone faces, including us, is that you can’t really look for Rust developers, because there are so few of them out there. So we ended up having to train the hires. We’ve had mixed experience. I run a cryptographic company. We build cryptographic protocols, proof systems. We take computation, and then we do a lot of processing on top of that computation. It becomes very intensive, both in terms of storage and computation. We use Rust for obvious reasons. When we are hiring for Rust developers, it is harder to find experienced developers because already proof systems are so complicated that we need specialized resources to talk to them. There is a heavy mathematical background that is necessary. Hence, we have to pick up developers who have math backgrounds, which again is a very small subset. So, that’s the challenge that we face currently.

E: Is this more of a cryptographic-related (hiring) problem?

A: At this point, I don’t even expect people to have cryptography experience. We’ll train them, but having the ability to pick up Rust so that they can perform at the level and, running production systems are different from POC level systems. So, that is a maturity problem we are seeing.

E: Anyone else wants to speak about the talent pool here? People who are looking for jobs can talk about the problems that they are facing when trying to join companies that are using Rust.

A: Find people who are good at cryptography and then train them on Rust because the learning curve for Rust is much lesser than cryptography.

A: We don’t do cryptography in the sense that we are doing encryption or decryption. We build protocols that are derived from cryptography, creating new kinds of hashes that are better for us, computationally speaking, is a small part of what we do. We take big crypto systems, protocols that are built on cryptography. That is straightforward.

How are Rust crypto libraries?

E: How are the underlying crypto libraries in Rust? Are they mature enough?

A: Yes, some of them, but, we are a niche. For example, we built hashing and rebuilt rescue for our purposes. So we have to start at a lower level and then we have to build.

E: Smart contracts are not specifically in assembly. Do you go down to assembly?

A: We would like to but we can’t. Rust is the closest we can get.

Rust in domain applications

E: Is there anyone who is building domain applications on Rust? A lot of use cases here are very low level system engineering kind of problems. I am wondering if there is anyone who is building general domain applications on Rust? So This is specific to a domain like insurance where the traditional advantages of Rust really shines through performance. Let’s say memory safety. Memory safety is if I’m building like a workflow business processes. Memory safety is not my primary concern. Again, I can build it on Java, and I can build it in Go. It’s not a big deal. I want to build features faster, rather than have them really be memory safe. Juspay UIs and workflows are all built on frameworks and on top of Haskell and PureScript. You can correct me if I am wrong.

A: Juspay is kind of built on Haskell. But the next part that we are trying out will be like a simple web service. So that is built on Rust.
The current fulfillment workflow in the product is built out for the India market, and that is built with Haskell. We want to expand to the global market and that is where Rust is coming. We are building this from ground up using Rust. So in this case, memory or performance itself may not have a big impact. Since we are expecting a large number of transactions and throughput, we want to take advantage of even the smallest benefits. There is definitely that trade-off that with using Rust which will somewhat slow down our development. This happens in the initial phase. The same behavior or argument also happens for something like Test Driven Development (TDD). People say that it is slow if you’re doing that, but eventually you ramp up.

E: You want a solid talent pool when it comes to types. Because you have an internal good talent pool when it comes to types and visibly complex backend systems, right? You have got a talent pool already.

A: So even in that case, we had around only two or three folks who had helped with the initial onboarding. After that, I think most of our people who work with Rust have been with Python or Java experience where the web service domain kind of comes into play. And for Rust, it isn’t that you need to know type system to code in Rust, but it forces you to use the type system and things along those lines. There is a bit of hit in terms of developer productivity when you are trying to negotiate with the compiler.

A: So I am from Jaguar Software. We are building a Fintech solution for lending. It is a general product, a domain based product, and it is an application software. We found that development in Rust gives a lot of return when you are going to provide the software to a lot of companies. We are going to run a number of servers and lambdas in the cloud. So the cloud cost will come down, and that will be the result. The stability of the software is too good as compared to anything else. But the development time is extremely slow. It was challenging to get started with a team of three people. Motivating these three people to go through the pain of learning was immense, primarily because we were all new developers, and we were not Rust developers. We found that a good balance for development to be faster is to go with Golang. But if you think that you are committed and later the product is really going to grow, then to start with Rust is a very good idea. If you are confident about the product and market, Rust is definitely good. Once you build one component in Rust, it never goes down. You can be rest assured that nothing wrong will happen ever at runtime in production.

Rust and C Foreign Function Interface (FFI)1

A: I am working on something where I am developing a library and I have to provide it to people who will be using it from Java, Python, C#, maybe, GO as well. And the issue is that with C, the Rust FFI is good. But when it comes to selling, managed languages like Java or .NET, then the FFI store is not so good, I hear.

A: With languages, such as a non-typical C FFI where it is required, you have to have your ecosystem-level tooling which needs to be able to do that communication. So those things are slowly being developed. With Python, there’s a library called PyOxidizer, which is supposed to allow writing Python and Rust code together. For web projects, you have wasm-pack². This allows you to instantly build your Rust code and compile it down to wasm, and generate all the TypeScript headers so that TypeScript types work with your wasm modules written in Rust. So those sorts of things exist. When you are using Rust, and if you want to do an FFI thing, then I recommend using wasm as an indirection layer. Wasm will also give you a very good amount of portability, because wasm interpreter exists on all platforms. And you can even deploy to embedded in some cases. If you are not going to do microcontrollers, it will be fine. If you want to talk to Java, then Java’s wasm APIs and Rust wasm APIs can integrate with each other, rather than them talking to each other directly via the CFI. Because getting CFI to work with each other is very hard. It gets very hard since you have to get a C FFI out of Rust, and then you get a C FFI into Java. Android developers have built their own tooling around it. They support Java and Kotlin. The Google team yesterday released a blog on how much Rust they have added to the Android ecosystem. They have almost made roughly 40% of the code base that was C and C++ to Rust, and they’re continuing to do so. And they plan to remove as much of the C and C++ components as possible, and move a lot of the unnecessary things that do not need performance to Kotlin or Java instead.

Rust and Golang

E: Google’s primary language is Go. I am curious how they are prioritizing Rust in favour?

A: I don’t think Google is very committed to a specific language. I don’t think any of the companies really care about the control over the language. Google also sponsors the Rust Foundation. At some point, there is a question of what is better for your purpose of writing applications. Now Google even has a second project called Carbon.

E: Think about it - first it was Java, and then they went to Kotlin.

A: And even now most of Google’s services are in C++.

E: What I am curious about is that they sponsored Go and it is competition to Rust.

A: Go came before Rust. Go was not really a competition to Rust specifically, but it was a competition to more of the Java and the ecosystem, which was for like domain specific tasks, to develop applications quickly, get them off the ground, have a consistent framework and have a lot of developers around it. So that is my perspective of Go. They might have a specific view. I do not think any of the companies are committed to specific languages because at some point, the sunk cost fallacy will also kick in, if you are sticking with something that is not working for you. Now they are building Carbon as well.

A: Large companies are generally like an agglomeration of different groups. Different groups have different incentives. So there might be lots, even in a company where there is a language that is mandated. I can give the example of Microsoft, for example. There is a lot of usage of Rust that is done from bottom-up. At some point, it becomes significant enough that the company mandated it. Like Google, Microsoft is a tools company. So there .NET. And there are competing groups within. At the end of the day, developers have some say in the management for building stuff, which is in big tech companies. Also, they understand that programming languages or tools are a means to an end, and hence supporting that will actually make their products better and have a longer life . So they have an interest in that part, and that is the reason why they will support multiple things.

Why is Rust fast?

A: I had a follow up question to probably everyone who is working on Rust. I have not worked on Rust so much is, but I’m looking at the some of the benchmarks. It seems the reasons why Rust is faster is probably because of the cache behaviors on CPUs, right? Is it because the memory footprint is less?

A: Any language that is smaller and generates less code will generally be faster. But that is only true to an extent.

A: Because of the compiler also that makes those decisions, which other languages have to make up at runtime, right? For example, there is no memory management in Go. So, that memory management, even if it is a small bit, that adds to the overall complexity of managed metrics. So if you are completely getting rid of that, you are getting rid of that code. Hence, cache behavior increases. You do not have to do things like type inference at runtime. It is even counterproductive to compare with Python or Ruby to something like Rust to some extent. This is because they are totally different paradigms of doing and thinking.

A: For every analyzer, we have pretty much bootstrapped everything. All the analyzers are written in the language they were built-in. So, Python analyzer uses Python. JavaScript analyzers use JavaScript, and this is for multiple reasons. This allows us to have in-house experience in those languages, because we are building analysis tools. So we need pretty high level of competence in those languages, so that we are able to analyze those languages. If you are finding your mistakes in your code, then we need to be at least aware of what those mistakes. We can only do that if we have some expertise. That was one decision. But the second decision is generally that language itself will have some compilers that will have parsing libraries for itself. All compilers these days come bootstrapped. Even TypeScript is written in TypeScript and JavaScript, mostly. So if you need those things, then you get it from the language itself. C++ compilers are written in C++. LLVM, for example, and Clang. Go’s compiler is in Go, and so on, so forth. So for us, it is not a matter of languages, but it is a matter of how a language behaves. Every language will have its trade-offs. There are very few languages that are entirely overlapping, or entirely competing. Even C and C++ don’t really compete. There is a reason Linux kernel never added C++ even though they had C. So they tried their best to find a language that they can adapt. And then we finally ended up with Rust, because of its safety guarantees, and none of the C alternatives were providing at the time.

A: So single binary thing. I am compiling to binary is fashionable again. I keep on seeing the fact.

A: Even monorepos have become very popular.

A: I referenced a talk where our clients filter API is in Rust, and it runs on basically all platforms. That talk addresses why they went to do that. And with Rust how they have achieved something which runs on iOS or Android.

A: Compute times are costly. Cloud compute is cheap. But the thing is wasting compute is expensive. We have extra computing. From Rust to Go, it won’t be a matter of difference. I don’t think anyone will care if their service was 10% difference other than Microsoft, Google, all the big companies whose difference in bills will be millions of dollars. But other people will not care. But the thing that they care about is what the safety guarantees of the ecosystem that they are using provides.

Rust and programmer productivity

A: On one side of the spectrum you have the programmer productivity and the other side is the language productivity. So something like Go is probably more towards programmer productivity and Rust is more towards the language.

A: There is a cost to everything. But with some languages, you will pay the cost upfront, with some you will pay later. Where you want to pay is something to consider.

A: If you look at a long enough timeframe, I think that Rust makes a lot more sense. If you want the software to last, and you plan to support it, maybe in a couple decades, then the trade-offs are very different.

A: For example, even if you think of embedded devices, you need minimum 10 years if you want to build something with it. Chips come with minimum 10 years guarantee, and you want to build something which runs on that device for that long and runs all the time stable.

Rust in embedded

E: The cost of replacing that device is also very high, right?

A: And you get locked into a very specific amount of compute. We are used to getting our compute scaled up year after year. Last year I bought a computer. Three years later, I will buy one, and it will make all my programs 10x faster. That is stopping with Moore’s law problems as well. And much more than that, that cannot happen with embedded. If we have shipped a television, and that has 1GB of RAM, and that is not going to scale up. It is not going to magically increase after five years of it being in production even if there are new products.

Business risks

E: One more aspect that I wanted to talk about is looking at this from a lens of risk. As a business, even as a startup or even a large company, you take bets and you take risk. In some sense, one can argue that diversity in some sense is part of your deployment plan, especially whatever you are going to deploy is going to stay around for a long time. Or if it is going to get invoked a lot, like serverless functions, it is probably better to have them written in Rust.

A: Generally that will be the case, but modern serverless architectures use runtimes like Interlake. AWS has its own runtime that they use to keep node, code hot, in memory. All the node code is always there in memory, because some level of memory is cheap. So they can just keep that cached. At that point, the start time of a binary can get overshadowed.

A: For example, document processing, maybe, running Optical Character Recognition (OCR) or Natural Language Processing (NLP) algorithms. You are doing that in serverless. In fact, I was talking to a startup that does exactly the same. And for the memory, when you are loading the page, you are also allocating memory for the words and the strings and everything that is there inside. So that memory performance matters. Because of the delta, even if it is 30%, and 30% on every invocation can be higher. So while the cold start time might be less, which means the runtime will still be more. And because the compute is charged on the basis of how many times we invoke and multiply by the compute time, that costs you. For example, let us say that you are building a document database for scanning all of the Supreme Court orders. Just one random example, or a big company has a lot of documents and they want to build a search engine or pre-processing on top of that. One is obviously the reliability of how good it is and how fast is it to operate, but also the other part that compute really matters is the number of invocations. So when I will look from a business point of view and look at languages, I think Rust makes sense. Why is it used in certain aspects is because every language is eventually kind of maps very nicely to some domains, where the trade-offs apply much better as compared.

A: Also the better exception handling. So the safety guarantees that every situation will be handled is just not there in other languages.

A: So, generally, Rust is best in case you want services that you don’t want to break. But sometimes we expect some exceptions, and Elixir and other languages like Erlang exists for that where you have something will go wrong, and it will just recover on its own.

A: That is what I meant by risk. The cause of failure is very hard. For example, Juspay does payment with Rust. The cost of that error rate increasing actually has a immediate impact on the bottom line. And that comes back to my other parameter, which is the number of invocation matters, because it is going to get invoked a lot of times and hence even a small increase in failure is going to have a direct hit on the bottomline, and also maybe reputation in some cases.

Rust adoption

E: I am trying to reconcile that with say, speed of development, at an early stage, like how soon is too soon?

A: The problem with Rust resources, or Rust developers and hiring and onboarding is finding Rust developers with enough experience is both hard and expensive. You you need to do a lot of interviews, and you need to find a lot of people. You need to figure out where can you find someone who has enough experience with Rust and also knows your domain. You need a mix of both in the person that you are trying to hire. If you already have that, then Rust is the obvious choice. But if you are constrained on those resources, then you might have to look if you are willing to invest in Rust. It will keep getting better with more and more people showing up for Rust, getting to know about Rust, and more people gaining experience with Rust. It will improve. But there is currently a cost where new companies will have to bear if they do not have in-house experience with Rust. They need to make sure that they don’t commit to something that they might not be able to maintain. If one guy is maintaining your entire Rust codebase, or critical tools have been maintained by one person, and he leaves for any reason, then you are practically not doing a very good job at managing those kinds of risks as well.

A: This works in larger companies because they are likely more resources to invest more into the larger projects. If those systems are maintained, then there is a pool of talent. The company benefits because products reflect the trade-offs that are made in those language ecosystems. Maybe something like Kubernetes is supported, and maybe a lot of the Apache projects are also supported for the same reason that the language are also supported.

A: For application development, one is onboarding and upskilling. Second is a churn. And the third is developer experience and productivity, which inherently some languages offer much more than low-level language abstractions that we can create.

A: Your go-to language should be Golang, because of all those factors pointed out. But if you are really thinking that you can invest upfront a lot of time and spend in training, maybe you have to hire non-Rust developers and make them developers of Rust.

A: In the best case scenario, if you already have Rust expertise, it can give you that edge than your competition probably. If everyone else is using something slow, and you are using something that is faster, then you inherently have an advantage. You are less resource constrained. For example, if a cloud solution provider is using Java, and you are using Rust, then Rust will give you much less resource usage, and you have a much greater margin and much greater bottom line. But that comes around to the question of what your resource allocation looks like.

E: It is like iPhone versus Android, since it is written in Objective-C.

A: It has that advantage of running much faster than an interpreted language.