My former argument? You might be confusing who you are talking to, since you answered to my first post in this thread.
You also seem to remember leftPad wrong. What happened there was that someone made a tiny library that did nothing but to pad a string. Something so trivial that any programmer should be able to do that within a minute. But still tens of thousands of projects, even large and important libraries, would rather add a whole dependency just to save writing a line of code. In fact, in most dependency management systems it requires more characters to add that dependency than to write that oneliner yourself.
The issue with leftpad was that the maintainer of that “library” was angry for unrelated reasons and pulled all his libraries, which then broke thousands of projects and libraries because leftpad wasn’t available any more.
My point was that everyone just relies on upstream doing their stuff and hardly anyone bothers to check that the code they include is actually doing what it should. And everyone just hopes that someone else already did their job of reviewing upstream, because they can’t be bothered to do it themselves.
A better example though would be Heartbleed. OpenSSL is used in everything. It’s one of the core libraries for modern online communication. Everyone and their grandma used it, most distros, all the cloud providers and so on. Everyone has been making money using the security that OpenSSL provides. Yet OpenSSL was massively underfunded with only one permanent developer who was also underpaid for what he was doing. And apparently nobody thoroughly reviewed the OpenSSL code. Somehow in version 1.0.1 someone made a mistake and added the Heartbleed bug. Stuff like that happens, nobody’s perfect, and if there’s only one person working on this, mistakes are bound to happen.
And then this massive security vulnerability just stayed in there for over two years, allowing anyone to read out whatever’s in the memory of any server using OpenSSL. Because nobody of the billions of people using OpenSSL daily actually reviewed and analysed their code. Because “so many people use OpenSSL, someone surely already reviewed it”.
Or take Log4Shell. That’s a bug that was so trivial it was even documented behaviour. To find this, someone wouldn’t even have had to review the code, just reviewing the documentation of Log4J would have been enough. And still this one was in production code for 8 years. For a library that’s used in almost every Java program.
Nobody reviews upstream.
If upstream makes a mistake, that mistake is in the code. And then everyone just happily consumes what they get.
And upstream is often just a random library thanklessly maintained by some dude in their spare time.
Edit: Just to prove my point: Think of your last big FOSS project that you worked on. Can you list every single dependency and every single transient dependency that your project uses? For each of these dependencies, do you know who maintains it and how many people work on each of these dependencies? Do you know if everyone of these people is qualified and trustworthy enough to put reliable and secure code in your project? Or do you, like everyone else, just hope that someone else made sure it’s all good?
You talk as though closed-source developers reviewed all the upstream code. The exact same problem exists with closed-source, except there isn’t even the possibility of reviewing all the code if you want to. At worst, the lack of review in FOSS projects is on par with closed-source projects. At best, it’s a much smaller problem .
That’s definitely a problem with every bit of code, that everyone relies on stuff they don’t or can’t review.
My point is that FOSS provides a false sense of security (“Millions of people use this library. Someone will already have reviewed it.”).
But the bigger issue is that FOSS is massively underfunded. If OpenSSL was for-profit, it would be a corporate project with dozens if not hundreds of developers. Nobody would buy a piece of core security infrastructure from a self-employed dude working away in his basement. That would be ridiculous to even think about that. And if this standard component was for-profit, even very low license fees would generate huge amounts of revenue (because it’s used in so many places) and this would allow for more developers to be employed.
And since it would be an actual thing that companies would actually buy, they’d demand that third-party security audits of the software would be done, like on any paid-for software that companies use. They’d also demand some SLA contracts that would hold this fictional for-profit OpenSSL accountable for vulnerabilities.
But since it’s FOSS, nobody cares. Companies just use it, nobody donates. It’s for free, so the decision to use it usually doesn’t even go through procurement and anything related to it. I tried to get my old company to donate to OpenSSL in the wake of Heartbleed, and the company said they don’t have a process to donate to something, so can’t be done.
So everyone just uses this little project created by one solitary hero and nobody pays for it. And so that dude works alone in his basement, with literally the online security of the whole world resting on his shoulders.
Luckily after Heartbleed a lot of large corporations started to donate to OpenSSL, but there are hundreds of other equally important projects that still nobody cares about. As seen e.g. with the .xz near miss.
If OpenSSL was for-profit, it would be a corporate project with dozens if not hundreds of developers
It seems like you don’t have a very broad exposure to closed source development. Corporations frequently have a skeleton crew working on a component or entire project. You might notice if you get escalated to development enough that it’s always like the same guy or two. It’s because they might only have a couple of guys working on it. Some companies will spend more on measures to obfuscate that reality than they would spend on actually developing. Certainly some corp closed source projects are that big, but so too are many open source projects.
Hell I’ve dealt with financial institutions using proprietary software that was abandoned by their vendor 15 years prior (came up because the software no longer worked with new stuff, and the institutions demanded wrapper software for new stuff to imitate the old stuff enough to keep using the unmaintained, unpatched, zero developer project).
I also don’t think companies are holding the proprietary vendors to quite the standard you imagine, certainly not automatically. By the same logic you propose for open source “someone else must have done it”, you also have that for big companies, if not more so. “Surely they have good security practices” or “it’s so popular someone must have done that”.
It seems like you don’t have a very broad exposure to closed source development.
Probably not. 15 years is not that long, what do I know, I’m just on senior expert level.
Companies run skeleton crews on crap products that don’t make money. Stuff they give away for free or that’s only used by legacy customers. Stuff they can’t shutdown because of contracts or because it still making a bit of money.
You might notice if you get escalated to development enough that it’s always like the same guy or two. It’s because they might only have a couple of guys working on it.
This is where your lack of knowledge about products like that shines through. It’s common to only get the same guy or two, because that’s the people designated (or willing) to talk to customers.
In real life, OpenSSL was run by a single person. That’s not a skeletton crew, that’s abandonment.
From what you are writing you aren’t a programmer and you haven’t worked in a software corporation before, but instead just extrapolate from your experiences with customer support.
Probably not. 15 years is not that long, what do I know, I’m just on senior expert level.
Longevity is not a guarantee of broad exposure. It may mean you have deep exposure, but making the rounds around the industry I can’t imagine maintaining such a universally optimistic picture of commercial management of software development.
Companies run skeleton crews on crap products that don’t make money.
Companies run skeleton crews on products when they think they can get away with it. Very high profile commercial projects with a lot of analyst attention may not be able to get away with it, but some surprisingly high profile projects without quite as much scrutiny get away with more than you would guess.
This is where your lack of knowledge about products like that shines through.
I’m speaking from familiarity with the provider side of things, wondering when a customer will catch on that they can’t seem to get that awesome support unless it’s the same guy as their peers get, and suspiciously unable to get decent support for a random week in June or something.
From what you are writing you aren’t a programmer and you haven’t worked in a software corporation before
Incorrect assumption on both counts. A few companies across a couple of decades and two of those companies extensively engaging with other companies on projects to get me some exposure to closed source development organizations even at some other companies.
Why do you seem convinced I can’t possibly be a software developer? Evidently your development career has given you one experience with a company that takes the task with a great deal of seriousness and I’ve seen that happen, but a lot of companies are not so diligent and either try to game things best they can either with like two people making git commits or an army of offshore developers that seem to quit within 6 months leaving little competency and plenty of opportunity for a bad actor to get in the door.
My former argument? You might be confusing who you are talking to, since you answered to my first post in this thread.
You also seem to remember leftPad wrong. What happened there was that someone made a tiny library that did nothing but to pad a string. Something so trivial that any programmer should be able to do that within a minute. But still tens of thousands of projects, even large and important libraries, would rather add a whole dependency just to save writing a line of code. In fact, in most dependency management systems it requires more characters to add that dependency than to write that oneliner yourself.
The issue with leftpad was that the maintainer of that “library” was angry for unrelated reasons and pulled all his libraries, which then broke thousands of projects and libraries because leftpad wasn’t available any more.
My point was that everyone just relies on upstream doing their stuff and hardly anyone bothers to check that the code they include is actually doing what it should. And everyone just hopes that someone else already did their job of reviewing upstream, because they can’t be bothered to do it themselves.
A better example though would be Heartbleed. OpenSSL is used in everything. It’s one of the core libraries for modern online communication. Everyone and their grandma used it, most distros, all the cloud providers and so on. Everyone has been making money using the security that OpenSSL provides. Yet OpenSSL was massively underfunded with only one permanent developer who was also underpaid for what he was doing. And apparently nobody thoroughly reviewed the OpenSSL code. Somehow in version 1.0.1 someone made a mistake and added the Heartbleed bug. Stuff like that happens, nobody’s perfect, and if there’s only one person working on this, mistakes are bound to happen.
And then this massive security vulnerability just stayed in there for over two years, allowing anyone to read out whatever’s in the memory of any server using OpenSSL. Because nobody of the billions of people using OpenSSL daily actually reviewed and analysed their code. Because “so many people use OpenSSL, someone surely already reviewed it”.
Or take Log4Shell. That’s a bug that was so trivial it was even documented behaviour. To find this, someone wouldn’t even have had to review the code, just reviewing the documentation of Log4J would have been enough. And still this one was in production code for 8 years. For a library that’s used in almost every Java program.
Nobody reviews upstream.
If upstream makes a mistake, that mistake is in the code. And then everyone just happily consumes what they get.
And upstream is often just a random library thanklessly maintained by some dude in their spare time.
Edit: Just to prove my point: Think of your last big FOSS project that you worked on. Can you list every single dependency and every single transient dependency that your project uses? For each of these dependencies, do you know who maintains it and how many people work on each of these dependencies? Do you know if everyone of these people is qualified and trustworthy enough to put reliable and secure code in your project? Or do you, like everyone else, just hope that someone else made sure it’s all good?
You talk as though closed-source developers reviewed all the upstream code. The exact same problem exists with closed-source, except there isn’t even the possibility of reviewing all the code if you want to. At worst, the lack of review in FOSS projects is on par with closed-source projects. At best, it’s a much smaller problem .
That’s definitely a problem with every bit of code, that everyone relies on stuff they don’t or can’t review.
My point is that FOSS provides a false sense of security (“Millions of people use this library. Someone will already have reviewed it.”).
But the bigger issue is that FOSS is massively underfunded. If OpenSSL was for-profit, it would be a corporate project with dozens if not hundreds of developers. Nobody would buy a piece of core security infrastructure from a self-employed dude working away in his basement. That would be ridiculous to even think about that. And if this standard component was for-profit, even very low license fees would generate huge amounts of revenue (because it’s used in so many places) and this would allow for more developers to be employed.
And since it would be an actual thing that companies would actually buy, they’d demand that third-party security audits of the software would be done, like on any paid-for software that companies use. They’d also demand some SLA contracts that would hold this fictional for-profit OpenSSL accountable for vulnerabilities.
But since it’s FOSS, nobody cares. Companies just use it, nobody donates. It’s for free, so the decision to use it usually doesn’t even go through procurement and anything related to it. I tried to get my old company to donate to OpenSSL in the wake of Heartbleed, and the company said they don’t have a process to donate to something, so can’t be done.
So everyone just uses this little project created by one solitary hero and nobody pays for it. And so that dude works alone in his basement, with literally the online security of the whole world resting on his shoulders.
Luckily after Heartbleed a lot of large corporations started to donate to OpenSSL, but there are hundreds of other equally important projects that still nobody cares about. As seen e.g. with the .xz near miss.
It seems like you don’t have a very broad exposure to closed source development. Corporations frequently have a skeleton crew working on a component or entire project. You might notice if you get escalated to development enough that it’s always like the same guy or two. It’s because they might only have a couple of guys working on it. Some companies will spend more on measures to obfuscate that reality than they would spend on actually developing. Certainly some corp closed source projects are that big, but so too are many open source projects.
Hell I’ve dealt with financial institutions using proprietary software that was abandoned by their vendor 15 years prior (came up because the software no longer worked with new stuff, and the institutions demanded wrapper software for new stuff to imitate the old stuff enough to keep using the unmaintained, unpatched, zero developer project).
I also don’t think companies are holding the proprietary vendors to quite the standard you imagine, certainly not automatically. By the same logic you propose for open source “someone else must have done it”, you also have that for big companies, if not more so. “Surely they have good security practices” or “it’s so popular someone must have done that”.
Probably not. 15 years is not that long, what do I know, I’m just on senior expert level.
Companies run skeleton crews on crap products that don’t make money. Stuff they give away for free or that’s only used by legacy customers. Stuff they can’t shutdown because of contracts or because it still making a bit of money.
This is where your lack of knowledge about products like that shines through. It’s common to only get the same guy or two, because that’s the people designated (or willing) to talk to customers.
In real life, OpenSSL was run by a single person. That’s not a skeletton crew, that’s abandonment.
From what you are writing you aren’t a programmer and you haven’t worked in a software corporation before, but instead just extrapolate from your experiences with customer support.
Longevity is not a guarantee of broad exposure. It may mean you have deep exposure, but making the rounds around the industry I can’t imagine maintaining such a universally optimistic picture of commercial management of software development.
Companies run skeleton crews on products when they think they can get away with it. Very high profile commercial projects with a lot of analyst attention may not be able to get away with it, but some surprisingly high profile projects without quite as much scrutiny get away with more than you would guess.
I’m speaking from familiarity with the provider side of things, wondering when a customer will catch on that they can’t seem to get that awesome support unless it’s the same guy as their peers get, and suspiciously unable to get decent support for a random week in June or something.
Incorrect assumption on both counts. A few companies across a couple of decades and two of those companies extensively engaging with other companies on projects to get me some exposure to closed source development organizations even at some other companies.
Again, support is not development. Experiences with support does not allow conclusions on development.
And having no experience in development doesn’t qualify you to make statements about development.
Why do you seem convinced I can’t possibly be a software developer? Evidently your development career has given you one experience with a company that takes the task with a great deal of seriousness and I’ve seen that happen, but a lot of companies are not so diligent and either try to game things best they can either with like two people making git commits or an army of offshore developers that seem to quit within 6 months leaving little competency and plenty of opportunity for a bad actor to get in the door.
Well, you said you only have experience from the outside.
That was feature creep galore. Ffs…