Thursday, July 7, 2022

Apple M1: Yet Another Last Chance for Linux on the Desktop

 This may be the last chance. No really.

This is not about advocacy of Linux on Apple's phenomenal M1 and its hardware.

But the M1 is so much better than other laptops at battery life, that it represents a fundamental leap in usability and quality.

M1 is an ARM instruction set running on an advanced TSMC fab process that currently other processors don't have access to but will. The process probably helps, but the real magic is almost certainly with the ARM instruction set.

AMD, Qualcomm, and maybe Intel will produce a response chip. But what OS would this chip run?

Here are the options:

1) Windows ARM: this was already attempted as was a total marketplace failure. Windows software is tightly coupled with x86 via gaming, backwards compatibility, and altready-purchased licensed software. Windows long built a moat to keep people in Windows, but that same moat keeps people from adopting ARM-based windows. 

2) Android: It can run on ARM, but android does not have the desktop applications and usability for desktop and laptop. It does have a Linux core however....

3) ChromeOS: a marketplace failure that never got traction. It does have a Linux core however...

4) Linux!

Linux has the cross-compiled apps already ready to go. It has kernels that are proven ARM compatible with all the cloud ARM VMs. It has the databases, browsers, and applications ready to go.

I have always wondered why Intel was so subservient to Microsoft, they should have had a linux-based OS that showcased their hardware to the maximum extent while Microsoft would always push hardware support to Service Packs or future releases. Now every processor vendor that wants to respond to Apple's M1 may need to do this. These cpu vendors have not just the size and financial resources, but the centralized influence and hardware relationships  to push adoption of new platforms and OSs.




Monday, February 8, 2021

Stages of Despair in Configuration

Configuration. It's just a text file right?

If you shrugged and agreed, you can probably stop reading. You are at Stage 0 of configuration despair and can live in blissful ignorance until you encounter higher stages. 

Morbidly curious or enlightened by experience? Please read on.

Here is an example of a order of configuration resolution from a "mature" framework: an excerpt from the Spring framework in Java: 

Spring Boot uses a very particular PropertySource order that is designed to allow sensible overriding of values. Properties are considered in the following order:

  1. Devtools global settings properties on your home directory (~/.spring-boot-devtools.properties when devtools is active).
  2. @TestPropertySource annotations on your tests.
  3. properties attribute on your tests. Available on @SpringBootTest and the test annotations for testing a particular slice of your application.
  4. Command line arguments.
  5. Properties from SPRING_APPLICATION_JSON (inline JSON embedded in an environment variable or system property).
  6. ServletConfig init parameters.
  7. ServletContext init parameters.
  8. JNDI attributes from java:comp/env.
  9. Java System properties (System.getProperties()).
  10. OS environment variables.
  11. RandomValuePropertySource that has properties only in random.*.
  12. Profile-specific application properties outside of your packaged jar (application-{profile}.properties and YAML variants).
  13. Profile-specific application properties packaged inside your jar (application-{profile}.properties and YAML variants).
  14. Application properties outside of your packaged jar (application.properties and YAML variants).
  15. Application properties packaged inside your jar (application.properties and YAML variants).
  16. @PropertySource annotations on your @Configuration classes.
  17. Default properties (specified by setting SpringApplication.setDefaultProperties)
Wow. Spring is intended to be near-universal in it's use cases for enterprise Java, and that "stack" of configuration value resolution isn't (solely) a product of exuberant coding. A large part of that likely evolved from spring being used in (10000s? 100000s? millions?)... a LOT of applications. 

To underline: "A SENSIBLE OVERRIDING OF VALUES"

A lot of that is java specific, but let's examine quickly some of the more universally true sources of configuration in that stack:

- a configuration file (yaml/json/properties/toml/xml) in the base dir of your app, or the home dir of your account, or in the common UNIX path for config settings (/etc/<your_app>). Oh, which one has precedence?

- when executing automated testing, you probably want some test-specific configuration to override base settings. 

- command line arguments: invocation of the program with directly specified arguments/configuration on the command line should override any other sources ... right? 

- OS environment value: oh right, those are like command line args. they should probably override or at least contribute, right?

- environment-specific configuration: similar to testing-only configuration, you'll have configuration specific to local dev, dev, stage, preprod, integration, acceptance, and production. While production config will likely only exist on one box, other environments may coexist on boxes so you don't waste infrastructure

- formats: I debated making this its own stage (or including it in stage 2), but configuration can be in properties files, json, yaml, toml, (ugh) xml, to say the least. All depending on programmer (usually) or end user (for "nice" programmers) preference. 

If you are nodding your head at all of that, you are at stage 1 of configuration despair. Uhoh, that's ONLY stage 1? 

Stage 2: <whispers>secretsssss</whispers>

Some of the more important parts of app configuration is secrets: passwords, keys, etc. Sure you can plop those in a text file somewhere, chmod 600, and rely on base UNIX security, but your client/company security, uh, overlords may have different requirements. It might be in aws secrets manager, in hashicorp vault, in some plain old LDAP or database, or a web service. 

Suddenly a source isn't a plain old text file or plain old cli args or plain old env vars. Your configuration subsystem may involve database drivers, webservice invocations, cloud interfaces, LDAP, decryption, etc. 

Welcome to Stage 2... retrieval of configuration uses arbitrary computing and really needs to be turing complete.  Wait, that's just stage 2? Yes, it gets worse.

Stage 3: Datatype Complexity

Wait, isn't all that above a big mound of complexity? Yes. Alas, there are so MANY kinds of complexity in IT.

An exhausted veteran of IT systems may notice that all we are really retrieving from configuration in the previous stages of despair implicitly are ... single values. Do computers ONLY take single values as input? Oh right. They take lists of values. They take map/dicts/associative arrays. Oh crap, they take entire tables of values (CSV), trees of values (deep json/yaml/xml). In the worst case: full on graphs of data objects. 


Don't believe me? Ever written ansible / terraform / chef? Heard of "configuration languages"? Ever templated your configuration files, and those template engines of course have an evaulation language. These all exist to help build out the configuration trees and graphs of data.

Oh crap, your configuration data which at a minimum are now trees of data, will probably have templating and/or cross-referencing values (basically almost an object graph). Your configuration has ... configuration. It's gone recursive. 

We're not done.

Unlike single-level single-value "precedence" you'll now have some tougher problems:

- If a configuration key's value is a map, does it overlay/combine/union its keys with the lower precedence maps (which can be very useful for DRY of common configuration / values)?
- If a configuration key's value is list, does it replace, append, union, intersect, etc with the lower precedence lists of values?
- For overlaid maps, how do you remove/undo/cancel the value of a lower-precedence map key that you don't want to be active? 

Welcome to Stage 3! Actually, between the recursive configuration and data combination, I probably could have split this one into more stages.

Still with me? Onto stage 4!

Stage 4: Filesystem overlays

This really is just a variant of all the previous three stages, but it does exist...

Ever heard of docker? You know how docker has that nice feature where a docker image can combine several "layers" of filesystems into an overall filesystem for your container? Know what that's doing?

Oh right, you're resolving configuration.

That's a POWERFUL technique.

But really it runs into all the previous stages' complexities: you'll have to determine the order of those filesystem overlays. Those "filesystems" may be subdirectories on the build machine, they may involve source control system pulls, sftp, secured systems, s3 buckets. 

Oh <deity or savior of your cultural preference>, what happens when there's a file in the same path in multiple overlaid filesystems? Generally you'd just do highest precedence file wins, but technically you may need to support diff/merges/parsing logic. What if you need to cancel out a file? How do you specify a "remove file" instruction?

Yes those are edge cases. But they exist....






Friday, June 26, 2020

The Recurring Tragedy of Orchestration

“Orchestration” is a recurring problem every IT professional of any experience will have encountered in multiple forms in their career, spanning dozens of tools and systems in a myriad of forms. It is an amorphous and multifaceted problem whose similarities with other systems always nags at me and has driven the sales of billions of dollars in flawed software that in the long term handcuffs an organization’s agility and freedom.

We’ve all seen this: a diagram with a series of boxes with arrows between them. Visio documents, viewgraphs, UML, workflows, process diagrams. Either they represent the abstract, or as documentation of specific processes (and usually out of date).

And we’ve all used a half-dozen IT tools that use the “boxes with arrows” as the fundamental conceptual paradigm: workflow engines and build systems in myriad forms, from UI tools that show the Visio-like interfaces, to configuration files that represent the same but without the UI.

The IT industry has reinvented this wheel over its entire existence, and it represents a massive inefficiency in the entire state-of-the-art. Why? We have seen dozens of these systems and they seem superficially extremely similar, and the work to produce definition, visualization, and execution has happened over and over and over again, often from a total rewrite. A massive amount of waste, except to software vendors.

Before going further, I want to emphasize that my conclusions should not be used to justify a “everything is a hammer” approach to one-orchestrator-to-rule-them-all, which will inevitably result in square pegs hammered into round holes. A lot of current orchestration software is purpose built due to practical constraints. The point of this … uh, paper … is to provide a plea for system evolution in the future to try to (ivory tower shining in the distance) work towards orchestration systems that are more flexible for multiple needs without the constant reinvention, conserves critical investment of code, and improves the ability of organizations to apply orchestration systems to their needs.

I think the essence of the problem, aside from the lessons of the tower of Babel, is both the abstract nature of the concept, but also the fact that the superficial impression of simplicity is actually much much much more complicated, and it runs into all the core problems of what makes computing and IT fundamentally complex.

The last part of that may question the value proposition of orchestrators. But orchestration systems undeniably have helped solve many problems effectively and have demonstrably tremendous value, which is why IT orgs the world over invest billions of dollars per year in them. They are a key means for visualizing and organizing the chaos and complexity in any IT org of non-trivial size. But these disparate orchestration systems always are immature in a large swathe of what an “ideal” orchestration system should be capable of.

The foundation of continuous delivery and integration of systems deployment is in its very nature orchestration.

Often orchestrators are constrained to only work in a specific ERP package (example: Documentum’s workflow component), or a specific purpose (Jenkins for builds, Spinnaker for “deployment”). Terminology is scattershot due to the multitude of applicable domains and the source of original design and responsibility.

Let’s take a shot at defining some key terms:

Concept/Definition: Orchestration

OK, so pretty important, what do we mean by this thing I’m complaining about?

Orchestration systems help coordinate lower-level computations and systems. It is “management” of computation, and by the word “management” I mean to directly correlate with human organizational management that MBA schools and a thousand books also have never properly solved. Orchestration systems are a desperate attempt to organize and coordinate the complex detailed work going on in an organization.

Almost all human organizations employ hierarchical structures: workers doing all the real work at the bottom, and then each level of management is dedicated to coordinating the level below them and communicating status (aka return values) to the level above. Managers of each domain of an organization will have specialized knowledge of necessary processes and procedures. The tiers of management “chunk” complexity and scope of responsibility into boxes and structure them so each hierarchy sees a higher level and overarching responsibility but delegates complexity. (see https://en.wikipedia.org/wiki/Span_of_control)

Are some of you thinking of Conway’s Law?

While invocation of Conway’s Law is usually in the context of disparagement and siloed data, the truth of Conway’s Law reflects a need by organizations to have software specific to their very specific needs. All levels of an organization define detailed procedures and processes and policies, and orchestration engines are basically the only software that will enable that to be somewhat structured and centrally managed. The universal application of Conway’s law is also why the constant total reinvention of orchestration is such a massive industry-wide failure.


Concept/Definition: Workflow / Process / Pipeline / Operation / Flow

Everyone that has used orchestration systems knows what they consist of: a group of defined “things” or capabilities it coordinates the execution of. Workflow systems have workflow definitions. Build systems have build definitions. Batch execution systems have their batch definitions.

These defined processes are usually configured in a non-binary format: XML, JSON, YAML, CSV, or other text format, or stored in a SQL database. Essentially these exist at the “metadata” level of IT orgs.

Often these consist of “subprocesses” for reuse or to implement “chunking”.


Concept/Definition: Execution / Run / Instance / Job

These are the individual instances of execution of a Process/Workflow, usually involving data such as the workflow definition (and likely version of that definition), the current status/state of the workflow such as what task or subprocess is currently working, and status values produced by already-executed steps, time the execution was initiated, parameters/context/data it was provided at start of execution, and error states encountered.

Concept/Definition: Task / Step / Unit

A hopefully-atomic unit of work from the perspective of the process/workflow. Either succeeds and allows the process/workflow to proceed, or there is an error which may disrupt or terminate the execution of a workflow. Basically serves to “chunk” the contained processing/execution into a simple success/failure state.

Execution Complexities: The Devil in the Details and Turing Machines

Unfortunately the selling of orchestration systems always starts at a sufficient level of ignorance and with a nice bit of snake oil: simple, easy flows of five or six boxes and arrows in a nice line. The salesman smiles and assures the IT management (who should know better) that the magic orchestrator will reduce all their complexities to these simple flows.

And I concede, relatively simple flows are THE sweet spot of orchestration, where the processes it handles, defines, visualizes, and executes are relatively easy. We probably all have stories of workflows doing crazy things like recursion or monumentally complex definitions that should have been broken down. As discussed in chunking and span of control, orchestrator workflow defs should orbit a central attactor of not-too-many-steps.


But… it isn’t. Every orchestration system when implemented starts hitting the hidden complexities in the processes it orchestrates. Unfortunately, to REALLY understand this you need to dip into the theoretical foundations of computing.

That four-box four-array straight line process is a very simple state machine with a single start point, single direction of flow, and one end state (DONE) or two (SUCCESS vs FAILURE). Theoretically this is a very very very simple form of computation, and make no mistake about it, orchestrators are just computation with prettier interfaces. Being able to run state machines like this is the very bottom basement of computational “power” that computational theory has mathematically proven. This isn’t an ivory tower thing either, this pecking order of “computational power” affects the very ability of orchestration to do useful work.

An orchestrator that can do these types of flow gets the golf clap and a nice pat on the head. Your orchestrator is executing very very very basic finite state machines. But there are much more difficult things an orchestrator will need to handle and those orchestrators with some maturity handle them to some degree:

- Branching: Wait, you need to DECIDE something? Yup, need some boolean evaluators and multiple routing. You're now a finite state machine with multiple paths coming out of points. This also is the gateway to needing something besides drag and drop shapes, and users come face-to-face with basic boolean math.

- Multiple start points: your process can start from several different initial states and begin the process at points that isn’t the “beginning”. Your process is becoming a more generalized form of finite state machine processing into a “directed acyclic graph” executor.

- Data/Memory/State/Storage: the “state” in “finite state machine” doesn’t imply memory or recording of values. Your execution of a process will invariably accumulate data and results that need to be stored. While this can be modelled with “finite” state machines of very large size, that isn’t useful. You instead are now, at minimum (probably), a pushdown automata/machine .

- Subroutines / Subflows: you define special-purpose processes that are “parameterized”, and want those processes to be invocable as single steps in overarching processes (using data from previous step results).

- Loops / Cycles: your processes may never halt and you are fully subject to the halting problem. Your state + your looping potential plus everything else means your process requires what is called a “Turing Machine”, which is what your computer is and what general purpose programming languages enable.

Execution Complexities: Error Handling, Recovery, and Rollback

Your basic salesperson snake oil barely touches aspects of error handling. In fact, most code written for programming languages, especially in enterprise systems, only superficially handles error. Really only very very mature software handles a wide variety of error conditions like database software. And that still fails in lots of ways.

A maturely defined process in an orchestration will have lots of variations in error reporting, handling, and recovery. When a lot of people think “my process/workflow doesn’t need looping” they aren’t thinking about errors and recovery, which almost any process definition will involve, and often involves REPEATING STEPS. Congratulations, your process definition has a loop, and is a big boy!

Reporting, recovery, retry, rollback, all of these are error handling techniques that EVERY orchestration software will have to support eventually. If your orchestration does not support it, it is a missing feature.

Execution Complexities: Distributed Execution and Parallelization


With the rise of cloud computing and managing fleets of servers, this has become more apparent, but orchestration has always involved the execution of processes across multiple machines and systems. That basic four-box snake oil the salesman is selling you? Guess what, that snake oil is hiding massively difficult computing problems in multi-phase commit, distributed computing, the CAP theorem, distributed state, parallel code execution, and many others.

The ideal orchestration system should handle this. Let me emphasize that it is very very unlikely that there is an orchestration system on the market that can handle all of that. The maturity of distributed systems, from Kafka, Zookeeper, Cassandra, etcd, Kubernetes, Hadoop, and all of those are still in their infancy in properly handling these operations.

Nevertheless, any orchestration system of maturity has features (they are probably broken for dozens of edge cases) that attempt to help users perform them.

Execution: Do Not Despair!


Readers at this point are probably despairing saying “just write custom code in a programming language”.

But we’ve already gotten a glimpse of the summit. Orchestrators SHOULD be able to do these things. I’ve seen a half-dozen workflow editors and the host of different Vizio boxes demonstrate that these thing should be doable in the (somewhat loosely defined) scope of what an orchestrator can orchestrate.

While parts of an orchestrator that handles some or all of these very difficult aspects of the process/workflow WILL require dropping down to full-power Turing complete programming languages, an orchestrator of good design should support this ability!

Perl’s philosophy of “more than one way to do it” will probably be needed here, especially given how orchestration has so many nebulous origins and different terms for “processing”.

Integration

A hot term from the years of message queues, n-tier architectures, SOAP/XML web services, and still applicable in the microservices trends, the ability to integrate disparate systems and pools of data are critical to an orchestration system, which often doubles as the integration layer funneling data between heterogenous systems.

Mature orchestration systems often can support a great deal of interfaces out of the box, but that support is lost each time orchestration is reinvented and often requires adaption over time or more layers of wrappers (often in front of other layers of wrappers written for previous integrations or point-to-point interfaces) that degrade performance.

Mature integration is also leveraged by software vendors for lock-in. Because the core of these integrations is often closed source, even in the open source orchestration frameworks, it becomes difficult to migration orchestration to improved forms.

User Interfaces

A critical differentiator between good orchestration software and a collection of motley scripts, programs, frameworks and other program/tool hodgepodges is the visualization that (hopefully) well-designed user interfaces provide.

These can provide facilities for visualizing the defined processes and adapting them (editing of workflows), as well as the state of their current executions, scheduling, triggering adhoc executions, visualization of failure states and means to reattempt/resume execution, debugging, history, and metrics.

In particular, the UI for showing state machines / DAGs / workflows / Visio-esque diagrams is not trivial, and often isn’t implemented until medium level maturity of orchestration systems. Attempts to standardize things via XML standards have generally failed due to typical subterfuge using embrace-extend incompatibilities by vendors, and the XML standards have been fairly use case centric. A “generally solved” framework or design for this would go a long way to helping the evolution of orchestration without the cycle of perpetual full UI rewrites.

The UI rewrite subproblem is of course related to the Recurring Tragedy of UI Frameworks constantly being reinvented in every programming language, OS, and more recently, every new HTML/JS framework, but that is another rant. At least with HTML/JS we have a universal rendered and a long lived base language of HTML/CSS and (sob) Javascript.

These host of valuable capabilities are generally lost with each successive reinvention until late in the maturity of the implementation. A lack of UI with good historical continuity that other applications have like spreadsheets, word processors, databases front ends, etc is a huge loss for the industry and practice.

A way forward?

Obviously the problem is extraordinarily difficult. The balkanization of ?hundreds? ?thousands? distinct software that have “semisolved” this problem, almost all overpromising/bragging they can do things of this type but not reporting the fine print (or even aware of the full scope of the problem) abound.

I think this will only begin to be solved in IDEs, another Tragic Tale of Reinvention (Oh do I weep for your passing Turbo Pascal, Delphi, and your ilk). But IDEs recently have started to coalesce and stay more persistent. JetBrains IDEs have shown real staying power, and Visual Studio is now fairly open and has lots of language support. And Eclipse… is still actively updated too. Many of these IDEs or dialects of them have the basic tools (workflow display and editing, etc) to do this.

Fundamental orchestration visualization tools in these IDEs, from project structuring, code organization, and debugging/testing visualization, kind of like what gdb and compilers provide, could provide the basis of structure to apply all the way up the food chain of tools. And IDEs almost always can be limited to function as purposeful tools with limited plugins and options.

Tuesday, October 15, 2019

Configuration inheritance and how it locks you in

So a quick definition of "configuration inheritance". It's not a new technique or recent revolution. It appears practically everywhere since the olden days. Basically, configuration inheritance is a base configuration/environment/set of settings that can be overridden by more specific cases/situations/users/etc.

So you start with, say this base environment, we'll say it's .rootrc:

   USERNAME=system
   MACHINE_NAME=some_server
   HOME_DIR=/home/system

But then you actually are running as user DUDE, and they inherit the base system's environment and decorate some other settings, in /home/DUDE/.bashrc file:

  USERNAME=DUDE
  HOME_DIR=/home/DUDE
  BACKUP_LOC=/mnt/backups/DUDE
 
 But then DUDE uses github, which applies and adds it's own settings from the .github file:

  USERNAME=dudeman
  GITPASSWORD=githubpassword
 
This example is just inside some UNIX environment, highlighting that this has existed since the late 1970s, and probably before then. It shows up all over programming, software, and systems management. To show you a more extreme example, here is the Spring settings hierarchy:

Spring Boot uses a very particular PropertySource order that is designed to allow sensible overriding of values. Properties are considered in the following order:
  1. Devtools global settings properties on your home directory (~/.spring-boot-devtools.properties when devtools is active).
  2. @TestPropertySource annotations on your tests.
  3. properties attribute on your tests. Available on @SpringBootTest and the test annotations for testing a particular slice of your application.
  4. Command line arguments.
  5. Properties from SPRING_APPLICATION_JSON (inline JSON embedded in an environment variable or system property).
  6. ServletConfig init parameters.
  7. ServletContext init parameters.
  8. JNDI attributes from java:comp/env.
  9. Java System properties (System.getProperties()).
  10. OS environment variables.
  11. A RandomValuePropertySource that has properties only in random.*.
  12. Profile-specific application properties outside of your packaged jar (application-{profile}.properties and YAML variants).
  13. Profile-specific application properties packaged inside your jar (application-{profile}.properties and YAML variants).
  14. Application properties outside of your packaged jar (application.properties and YAML variants).
  15. Application properties packaged inside your jar (application.properties and YAML variants).
  16. @PropertySource annotations on your @Configuration classes.
  17. Default properties (specified by setting SpringApplication.setDefaultProperties).
... I can hear Harry Caray going "holy cow". That hierarchy has lots of sensible features that involve testing, dev vs stage vs production environment settings, ability to do adhoc overrides, defaults, etc. There's reasonable justification for such a complicated stack, including evolution over time and hard learned lessons.

Really what this data pattern is a series of maps / dictionaries / key-value files that are then stacked upon each other, and the lookup then starts at the top of the stack and descends until it finds a value of a given key.

You see it in the above examples, in kubernetes/docker image overlays, in practically every infrastructure management tool, and dozens of other places.

Why? this is a good tool for centralizing common properties at one level, and overriding in specific situations, in order to manage settings across a variety of environments, or machines, or situations, or use cases.

It also enables a key aspect of software development and systems management: iterative/evolutionary development, as new requirements, systems, interfaces are added to something, or in the case of pure software development, successive releases to support increasing complexity.

But what seems like a solid foundation really is a teetering tower once you get five layers of configuration inheritance. What provides the illusion of management simplicity completely falls apart when significant reconfiguration occurs at the lower levels of the "stack of maps". And that is a key way in how organizations are locked into systems and software packages.

Here's the key feature that almost every one of these configuration inheritance / stacked maps lacks: a basic meta-mapping feature that explains where each value comes from. Using our first example of unix environment variables, which are displayed in any "context" by using the env command:

    USERNAME=dudeman
    GITPASSWORD=githubpassword
    HOME_DIR=/home/DUDE
    MACHINE_NAME=some_server
    BACKUP_LOC=/mnt/backups/DUDE

So my suggestion is that a key facility should be the metaenv command:

    .github::USERNAME=dudeman
    .github::GITPASSWORD=githubpassword
    .bashrc::HOME_DIR=/home/DUDE
    .rootrc::MACHINE_NAME=some_server
    .bashrc::BACKUP_LOC=/mnt/backups/DUDE

This command will show the additional information about which of the three files (in this example) actual values were assigned or possibly overridden, so that if there is a problem, bug, or unexpected value, someone can walk up the chain.

Unfortunately, once you get to the concept of multiple users, applications, servers, environments, etc etc, you form a huge tree of different inheriting situations, much like the huge inheritance hierarchies of OOP software and other situations.

The real problem with that is that trees are very hard to visualize with tools, especailly command line tools. But they are possible, and that is another class of tool that also does not accompany these systems, but should.

Because, should you apply a change to a file that affects a lot of other files that inherit from it, it would be nice to visualize all the impacted "downstream" files, and ALSO which downstream files are NOT affected because they override the values you changed.

Monday, October 7, 2019

NextStep OS fits in processor cache.

Back in the day of me buying a 486 PC for my college days, I remember thumbing through Computer Shopper. Somewhere in those days someone took the cache memory for processors and made a whole computer memory out of them. It was a lot more expensive!

But since those days and with the massive growth in silicon for L2/L3 caches of modern processors, I reflect on those days of operating systems and wonder "what if we could fit our OS and its apps in cache"?

Old Windows 3.1 or MS-DOS was not a pleasant experience compared to modern operating systems, almost no one can deny that. But NeXTSTEP? The color variants that came out after NeXT gave up on its black and white cubes... that is basically a modern UI, and arguably better than modern Linux GUIs.

https://www.youtube.com/watch?v=TIrTh80Z8jw

Old NeXT cubes ran on 16-64 megabytes of RAM and 25-33 megahertz of CPU, and 400 MB to 1 GB of disk. I can't find good numbers on the color variant that ran on x86, but I think our machines at college ran that very nicely on about 32 megabytes of RAM.

The prime curve of Moore's Law has produced quite the hardware bounty. Old timers like me bemoan a seemingly bloated software layer that has grown over the decades. The current Ryzen processors boast 80 megabytes of cache, and of course RAM is dollars a gigabyte, and SSD storage is getting reasonable.

Holy crap, NeXT would run entirely in cache. OK, swap might start getting into RAM. Modern RAM comes in the tens of gigabytes, much bigger than the hard disk on those machines. It basically looks like a modern OS, with a bit less aliasing on the fonts and icons. Productivity app set is practically identical. It had TCPIP networking, an HTML 1.0 web browser, spreadsheets, mail, etc.

With the end of big leaps from Moore's Law, if we want more performance, we'll need to optimize the software stack. I think a really cool project would be to start with the basis of NeXTStEP circa 1995-1997 and very carefully add in only what is needed. I remember the Mach kernel being buggy, but that can be swapped out.

Thursday, December 14, 2017

So many "what is a microservice" posts...

First off: I am not a zealotous proponent of maximum microservices. There are problems galore in constantly changing services that are also interdependent on each other. But the sheer number of bad microservice posts trying to detail what they are and why they have cache is astounding. It isn't that hard, people.

Most of these bad introductions starts at the software/component/design level, trying to compare architectural diagrams or styles, and contrasting with "monolith" architectures. ...Totally misses the main point of microservices that drives their benefits and values. Microservices fundamentally don't

Microservices are small in the sense that they are easily integrated atomic unit into modern cloud/devops/infrastructure-as-code/DatacenterOS computing and continuous delivery platforms. That is a microservice: whether it's a small VM, a docker image, jails/LXC, whatever. They are easily provisioned, deployed, relocated, and redeployed on these platforms.

Microservices plug into continuous delivery and version control systems with automated testing and immediate deployment to at least qa/staging environments once the build passes (and rapid deployment to production once the tests pass). Because they are "small", the incremental builds are relatively quick (an hour or two tops) even though they use lots of modern code coverage, code quality, code formatting, automated testing, automated deployment, and similar tasks beyond simple compilation.

Here are things that start to make your code not-microservice:

- proprietary software: There's a reason microservices are built to the wazoo on open source software. License servers, license acquisition, audits... always introduce a screeching halt to devops. Sure that can be bolted on, but there often are even more insidious things that get in the way: poor command line tooling, over-reliance on UI configuration, hidden "consultant bombs" and obfuscated documentation that prevent you from efficiently making a continuous integration pipeline or versioning your code with it.

Part of the small code and huge numbers of servers in microservices actively defends against the insidious influence of proprietary software salespeople: Sure organizations will buy 10 or even 100 licenses. But 10,000 Websphere licenses? Forget it. Microservices by preempting proprietary software enables scaling down the line. A definite feature in most circumstances.

- manual deployments: I get you might need a production approval process in many situations. But if each deployment requires some ticket to be filed and fingers drummed on a desk while you wait for some long list of manual tasks to be performed... forget about it.

- hardware starvation: while hardware costs shouldn't be ignored, if your organization can't deploy a $5,000 server without $50,000 of meetings, approvals, and documentation, or if your "cloud" is constantly running out of servers and disk space to power your software, then you can't do microservices. The devops platforms it depends on require sensibly budgeted hardware that can handle new services. That doesn't mean a small piddly service needs a quad-core processor and 4 gigs of ram. It means that the autodeployment step in the continuous build only fails due to hardware shortage once every few months, and that is fixed within a day, tops.

If your huge 1,000,000 line "monolith" legacy service can be fit into the above: containerized, continuously delivered, and deployed within a few hours of pull request or some similar software change, then congratulations: you are a elite microservice team.

- multiple organizations served in a codebase: Per Conway's law, your software WILL align with your business organization. Some of the problems of the big monolithic ERP software was to couple disparate business divisions in the same system. So programmers maintaining software couldn't find the parts of the monolith they needed when they were inevitably aligned with some business function. Microservices help keep enterprises' custom software isolated in specific functions that can be documented and maintained more easily. If some function is to be reused (but inevitably... slightly... different) across organizations, microservices should implement the shared model, and then write dependent services for the specific organization variants.

Thursday, August 20, 2015

Cloud and Devops: eliminating an entire layer of assholes

I've been in quite a few gigantic enterprises. Medical, Insurance, Retail, Government, you name it.

Without exception, in every one of these organizations the infrastructure teams function as blights upon innovation and adaptability, and impose unreasonable costs and delays. Infrastructure is not simple, I will grant... and this is not revelatory news to anyone in the industry.

It used to be, oh, say, a decade ago that we heard only rumors of Google, Facebook, Amazon, and other firms with dynamic, adaptive, cost-effective, anticipatory, and plentiful infrastructure for testing, prototyping, and production. No nightmares of 8 extra GB of RAM costing $1,000, or a simple PC server carrying $10,000 / year in upkeep chargebacks.

Once rumors, AWS / Rackspace / Google Compute and a host of other cloud providers have made that a much more stark contrast to anyone mired in internal infrastructure. Once things like Chef and other automated provisioning come into play, the contrast becomes even more pronounced.

Again, none of this is revelatory. Clouds are better cost, more adaptive, efficient, allow you to discover new technologies and make those practical to your job, group, and enterprise. Anyone who has done a cloud-enabled project has experienced the nirvana... But only now when we are being cast back into the rats nest of mandated internal infrastructure use do I recall all those meetings and emails and total clusterfucks:

  • your RAM is too big, we can't support more than 8GB. Oh, you want 4GB? That will be $1000/year extra
  • oh, you need another 100 MB of disk space, that will be $1000/year. No we won't provision you extra space, you have to crash in production and then raise a ticket to get us to do anything.
  • Let's schedule 10 meetings to discuss minor hardware spec increases that should be 1/100 of the labor waste of the meetings
  • Hardware guys bloviate about what your application's hardware spec requirements are, even though they have NO IDEA about what it does or its user base
  • CPU? You like 386s right? 
  • Oh, your VM is colocated with other extremely high use VMs? Too bad, so sad.
  • SSD? What is that?
  • Cloud? We have an internal cloud. No we don't have elastic storage or API provisioning. We can get your new VM up in 5 months rather than 1 year!
  • Accounting... ah, the accounting. "It's free... if you can get it". "It has no upkeep cost, but it costs a ton to setup". "Will the cost drop over time as Moore's Law marches on? NO". 
I am so fucking sick of it. Thus I DECLARE:

The #1 Feature of Clouds is the Elimination of an Entire Layer of Assholes From Your Life

That's right. #1 feature by far. No one from our cloud provider has ever sent us threatening emails yanking our RAM size, or down-specced our machines without consultation, or dragged their feet to provide new machines, disk space, CPU, whatever. Or spent weeks ignoring tickets behind ticket walls. No one scheduling bloviating meetings and handing down arbitrary new policies from above. No meddling in your application architecture. No complaints to the CEO if you decide to use someone else's infrastructure. None of it.

Not just any arbitrary layer of assholes. Infrastructure lends itself to controlling and sociopathic shitheads. They get to ooh and aah over high-spec machines, they get large lump-sum budget line items, and, best of all, they get to lord over the distribution of those resources to the people that need them, and demand asskissing and tribute. This is a particularly obnoxious layer of entitled, intractable, petulant assholes you are extricating from your life.

What is the #1 downside of our collective profession? Stress. Stress that causes measurable, documented deleterious health effects. If you can identify something that significantly reduces that stress, and I argue clouds can do that, then it should be a priority: personally, professionally, and industry-wide.