Apple M1: Yet Another Last Chance for Linux on the Desktop

 This may be the last chance. No really.

This is not about advocacy of Linux on Apple's phenomenal M1 and its hardware.

But the M1 is so much better than other laptops at battery life, that it represents a fundamental leap in usability and quality.

M1 is an ARM instruction set running on an advanced TSMC fab process that currently other processors don't have access to but will. The process probably helps, but the real magic is almost certainly with the ARM instruction set.

AMD, Qualcomm, and maybe Intel will produce a response chip. But what OS would this chip run?

Here are the options:

1) Windows ARM: this was already attempted as was a total marketplace failure. Windows software is tightly coupled with x86 via gaming, backwards compatibility, and altready-purchased licensed software. Windows long built a moat to keep people in Windows, but that same moat keeps people from adopting ARM-based windows. 

2) Android: It can run on ARM, but android does not have the desktop applications and usability for desktop and laptop. It does have a Linux core however....

3) ChromeOS: a marketplace failure that never got traction. It does have a Linux core however...

4) Linux!

Linux has the cross-compiled apps already ready to go. It has kernels that are proven ARM compatible with all the cloud ARM VMs. It has the databases, browsers, and applications ready to go.

I have always wondered why Intel was so subservient to Microsoft, they should have had a linux-based OS that showcased their hardware to the maximum extent while Microsoft would always push hardware support to Service Packs or future releases. Now every processor vendor that wants to respond to Apple's M1 may need to do this. These cpu vendors have not just the size and financial resources, but the centralized influence and hardware relationships  to push adoption of new platforms and OSs.

Stages of Despair in Configuration

Configuration. It's just a text file right?

If you shrugged and agreed, you can probably stop reading. You are at Stage 0 of configuration despair and can live in blissful ignorance until you encounter higher stages. 

Morbidly curious or enlightened by experience? Please read on.

Here is an example of a order of configuration resolution from a "mature" framework: an excerpt from the Spring framework in Java: 

Spring Boot uses a very particular PropertySource order that is designed to allow sensible overriding of values. Properties are considered in the following order:

1. Devtools global settings properties on your home directory (~/.spring-boot-devtools.properties when devtools is active).
2. @TestPropertySource annotations on your tests.
3. properties attribute on your tests. Available on @SpringBootTest and the test annotations for testing a particular slice of your application.
4. Command line arguments.
5. Properties from SPRING_APPLICATION_JSON (inline JSON embedded in an environment variable or system property).
6. ServletConfig init parameters.
7. ServletContext init parameters.
8. JNDI attributes from java:comp/env.
9. Java System properties (System.getProperties()).
10. OS environment variables.
11. A RandomValuePropertySource that has properties only in random.*.
12. Profile-specific application properties outside of your packaged jar (application-{profile}.properties and YAML variants).
13. Profile-specific application properties packaged inside your jar (application-{profile}.properties and YAML variants).
14. Application properties outside of your packaged jar (application.properties and YAML variants).
15. Application properties packaged inside your jar (application.properties and YAML variants).
16. @PropertySource annotations on your @Configuration classes.
17. Default properties (specified by setting SpringApplication.setDefaultProperties)

Wow. Spring is intended to be near-universal in it's use cases for enterprise Java, and that "stack" of configuration value resolution isn't (solely) a product of exuberant coding. A large part of that likely evolved from spring being used in (10000s? 100000s? millions?)... a LOT of applications. 

To underline: "A SENSIBLE OVERRIDING OF VALUES"

A lot of that is java specific, but let's examine quickly some of the more universally true sources of configuration in that stack:

- a configuration file (yaml/json/properties/toml/xml) in the base dir of your app, or the home dir of your account, or in the common UNIX path for config settings (/etc/<your_app>). Oh, which one has precedence?

- when executing automated testing, you probably want some test-specific configuration to override base settings. 

- command line arguments: invocation of the program with directly specified arguments/configuration on the command line should override any other sources ... right? 

- OS environment value: oh right, those are like command line args. they should probably override or at least contribute, right?

- environment-specific configuration: similar to testing-only configuration, you'll have configuration specific to local dev, dev, stage, preprod, integration, acceptance, and production. While production config will likely only exist on one box, other environments may coexist on boxes so you don't waste infrastructure

- formats: I debated making this its own stage (or including it in stage 2), but configuration can be in properties files, json, yaml, toml, (ugh) xml, to say the least. All depending on programmer (usually) or end user (for "nice" programmers) preference. 

If you are nodding your head at all of that, you are at stage 1 of configuration despair. Uhoh, that's ONLY stage 1? 

Stage 2: <whispers>secretsssss</whispers>

Some of the more important parts of app configuration is secrets: passwords, keys, etc. Sure you can plop those in a text file somewhere, chmod 600, and rely on base UNIX security, but your client/company security, uh, overlords may have different requirements. It might be in aws secrets manager, in hashicorp vault, in some plain old LDAP or database, or a web service. 

Suddenly a source isn't a plain old text file or plain old cli args or plain old env vars. Your configuration subsystem may involve database drivers, webservice invocations, cloud interfaces, LDAP, decryption, etc. 

Welcome to Stage 2... retrieval of configuration uses arbitrary computing and really needs to be turing complete.  Wait, that's just stage 2? Yes, it gets worse.

Stage 3: Datatype Complexity

Wait, isn't all that above a big mound of complexity? Yes. Alas, there are so MANY kinds of complexity in IT.

An exhausted veteran of IT systems may notice that all we are really retrieving from configuration in the previous stages of despair implicitly are ... single values. Do computers ONLY take single values as input? Oh right. They take lists of values. They take map/dicts/associative arrays. Oh crap, they take entire tables of values (CSV), trees of values (deep json/yaml/xml). In the worst case: full on graphs of data objects. 


Don't believe me? Ever written ansible / terraform / chef? Heard of "configuration languages"? Ever templated your configuration files, and those template engines of course have an evaulation language. These all exist to help build out the configuration trees and graphs of data.

Oh crap, your configuration data which at a minimum are now trees of data, will probably have templating and/or cross-referencing values (basically almost an object graph). Your configuration has ... configuration. It's gone recursive. 

We're not done.

Unlike single-level single-value "precedence" you'll now have some tougher problems:

- If a configuration key's value is a map, does it overlay/combine/union its keys with the lower precedence maps (which can be very useful for DRY of common configuration / values)?
- If a configuration key's value is list, does it replace, append, union, intersect, etc with the lower precedence lists of values?
- For overlaid maps, how do you remove/undo/cancel the value of a lower-precedence map key that you don't want to be active? 

Welcome to Stage 3! Actually, between the recursive configuration and data combination, I probably could have split this one into more stages.

Still with me? Onto stage 4!

Stage 4: Filesystem overlays

This really is just a variant of all the previous three stages, but it does exist...

Ever heard of docker? You know how docker has that nice feature where a docker image can combine several "layers" of filesystems into an overall filesystem for your container? Know what that's doing?

Oh right, you're resolving configuration.

That's a POWERFUL technique.

But really it runs into all the previous stages' complexities: you'll have to determine the order of those filesystem overlays. Those "filesystems" may be subdirectories on the build machine, they may involve source control system pulls, sftp, secured systems, s3 buckets. 

Oh <deity or savior of your cultural preference>, what happens when there's a file in the same path in multiple overlaid filesystems? Generally you'd just do highest precedence file wins, but technically you may need to support diff/merges/parsing logic. What if you need to cancel out a file? How do you specify a "remove file" instruction?

Yes those are edge cases. But they exist....

The Recurring Tragedy of Orchestration

“Orchestration” is a recurring problem every IT professional of any experience will have encountered in multiple forms in their career, spanning dozens of tools and systems in a myriad of forms. It is an amorphous and multifaceted problem whose similarities with other systems always nags at me and has driven the sales of billions of dollars in flawed software that in the long term handcuffs an organization’s agility and freedom.

We’ve all seen this: a diagram with a series of boxes with arrows between them. Visio documents, viewgraphs, UML, workflows, process diagrams. Either they represent the abstract, or as documentation of specific processes (and usually out of date).

And we’ve all used a half-dozen IT tools that use the “boxes with arrows” as the fundamental conceptual paradigm: workflow engines and build systems in myriad forms, from UI tools that show the Visio-like interfaces, to configuration files that represent the same but without the UI.

The IT industry has reinvented this wheel over its entire existence, and it represents a massive inefficiency in the entire state-of-the-art. Why? We have seen dozens of these systems and they seem superficially extremely similar, and the work to produce definition, visualization, and execution has happened over and over and over again, often from a total rewrite. A massive amount of waste, except to software vendors.

Before going further, I want to emphasize that my conclusions should not be used to justify a “everything is a hammer” approach to one-orchestrator-to-rule-them-all, which will inevitably result in square pegs hammered into round holes. A lot of current orchestration software is purpose built due to practical constraints. The point of this … uh, paper … is to provide a plea for system evolution in the future to try to (ivory tower shining in the distance) work towards orchestration systems that are more flexible for multiple needs without the constant reinvention, conserves critical investment of code, and improves the ability of organizations to apply orchestration systems to their needs.

I think the essence of the problem, aside from the lessons of the tower of Babel, is both the abstract nature of the concept, but also the fact that the superficial impression of simplicity is actually much much much more complicated, and it runs into all the core problems of what makes computing and IT fundamentally complex.

The last part of that may question the value proposition of orchestrators. But orchestration systems undeniably have helped solve many problems effectively and have demonstrably tremendous value, which is why IT orgs the world over invest billions of dollars per year in them. They are a key means for visualizing and organizing the chaos and complexity in any IT org of non-trivial size. But these disparate orchestration systems always are immature in a large swathe of what an “ideal” orchestration system should be capable of.

The foundation of continuous delivery and integration of systems deployment is in its very nature orchestration.

Often orchestrators are constrained to only work in a specific ERP package (example: Documentum’s workflow component), or a specific purpose (Jenkins for builds, Spinnaker for “deployment”). Terminology is scattershot due to the multitude of applicable domains and the source of original design and responsibility.

Let’s take a shot at defining some key terms:

Concept/Definition: Orchestration

OK, so pretty important, what do we mean by this thing I’m complaining about?

Orchestration systems help coordinate lower-level computations and systems. It is “management” of computation, and by the word “management” I mean to directly correlate with human organizational management that MBA schools and a thousand books also have never properly solved. Orchestration systems are a desperate attempt to organize and coordinate the complex detailed work going on in an organization.

Almost all human organizations employ hierarchical structures: workers doing all the real work at the bottom, and then each level of management is dedicated to coordinating the level below them and communicating status (aka return values) to the level above. Managers of each domain of an organization will have specialized knowledge of necessary processes and procedures. The tiers of management “chunk” complexity and scope of responsibility into boxes and structure them so each hierarchy sees a higher level and overarching responsibility but delegates complexity. (see https://en.wikipedia.org/wiki/Span_of_control)

Are some of you thinking of Conway’s Law?

While invocation of Conway’s Law is usually in the context of disparagement and siloed data, the truth of Conway’s Law reflects a need by organizations to have software specific to their very specific needs. All levels of an organization define detailed procedures and processes and policies, and orchestration engines are basically the only software that will enable that to be somewhat structured and centrally managed. The universal application of Conway’s law is also why the constant total reinvention of orchestration is such a massive industry-wide failure.


Concept/Definition: Workflow / Process / Pipeline / Operation / Flow

Everyone that has used orchestration systems knows what they consist of: a group of defined “things” or capabilities it coordinates the execution of. Workflow systems have workflow definitions. Build systems have build definitions. Batch execution systems have their batch definitions.

These defined processes are usually configured in a non-binary format: XML, JSON, YAML, CSV, or other text format, or stored in a SQL database. Essentially these exist at the “metadata” level of IT orgs.

Often these consist of “subprocesses” for reuse or to implement “chunking”.


Concept/Definition: Execution / Run / Instance / Job

These are the individual instances of execution of a Process/Workflow, usually involving data such as the workflow definition (and likely version of that definition), the current status/state of the workflow such as what task or subprocess is currently working, and status values produced by already-executed steps, time the execution was initiated, parameters/context/data it was provided at start of execution, and error states encountered.

Concept/Definition: Task / Step / Unit

A hopefully-atomic unit of work from the perspective of the process/workflow. Either succeeds and allows the process/workflow to proceed, or there is an error which may disrupt or terminate the execution of a workflow. Basically serves to “chunk” the contained processing/execution into a simple success/failure state.

Execution Complexities: The Devil in the Details and Turing Machines

Unfortunately the selling of orchestration systems always starts at a sufficient level of ignorance and with a nice bit of snake oil: simple, easy flows of five or six boxes and arrows in a nice line. The salesman smiles and assures the IT management (who should know better) that the magic orchestrator will reduce all their complexities to these simple flows.

And I concede, relatively simple flows are THE sweet spot of orchestration, where the processes it handles, defines, visualizes, and executes are relatively easy. We probably all have stories of workflows doing crazy things like recursion or monumentally complex definitions that should have been broken down. As discussed in chunking and span of control, orchestrator workflow defs should orbit a central attactor of not-too-many-steps.


But… it isn’t. Every orchestration system when implemented starts hitting the hidden complexities in the processes it orchestrates. Unfortunately, to REALLY understand this you need to dip into the theoretical foundations of computing.

That four-box four-array straight line process is a very simple state machine with a single start point, single direction of flow, and one end state (DONE) or two (SUCCESS vs FAILURE). Theoretically this is a very very very simple form of computation, and make no mistake about it, orchestrators are just computation with prettier interfaces. Being able to run state machines like this is the very bottom basement of computational “power” that computational theory has mathematically proven. This isn’t an ivory tower thing either, this pecking order of “computational power” affects the very ability of orchestration to do useful work.

An orchestrator that can do these types of flow gets the golf clap and a nice pat on the head. Your orchestrator is executing very very very basic finite state machines. But there are much more difficult things an orchestrator will need to handle and those orchestrators with some maturity handle them to some degree:

- Branching: Wait, you need to DECIDE something? Yup, need some boolean evaluators and multiple routing. You're now a finite state machine with multiple paths coming out of points. This also is the gateway to needing something besides drag and drop shapes, and users come face-to-face with basic boolean math.

- Multiple start points: your process can start from several different initial states and begin the process at points that isn’t the “beginning”. Your process is becoming a more generalized form of finite state machine processing into a “directed acyclic graph” executor.

- Data/Memory/State/Storage: the “state” in “finite state machine” doesn’t imply memory or recording of values. Your execution of a process will invariably accumulate data and results that need to be stored. While this can be modelled with “finite” state machines of very large size, that isn’t useful. You instead are now, at minimum (probably), a pushdown automata/machine .

- Subroutines / Subflows: you define special-purpose processes that are “parameterized”, and want those processes to be invocable as single steps in overarching processes (using data from previous step results).

- Loops / Cycles: your processes may never halt and you are fully subject to the halting problem. Your state + your looping potential plus everything else means your process requires what is called a “Turing Machine”, which is what your computer is and what general purpose programming languages enable.

Execution Complexities: Error Handling, Recovery, and Rollback

Your basic salesperson snake oil barely touches aspects of error handling. In fact, most code written for programming languages, especially in enterprise systems, only superficially handles error. Really only very very mature software handles a wide variety of error conditions like database software. And that still fails in lots of ways.

A maturely defined process in an orchestration will have lots of variations in error reporting, handling, and recovery. When a lot of people think “my process/workflow doesn’t need looping” they aren’t thinking about errors and recovery, which almost any process definition will involve, and often involves REPEATING STEPS. Congratulations, your process definition has a loop, and is a big boy!

Reporting, recovery, retry, rollback, all of these are error handling techniques that EVERY orchestration software will have to support eventually. If your orchestration does not support it, it is a missing feature.

Execution Complexities: Distributed Execution and Parallelization


With the rise of cloud computing and managing fleets of servers, this has become more apparent, but orchestration has always involved the execution of processes across multiple machines and systems. That basic four-box snake oil the salesman is selling you? Guess what, that snake oil is hiding massively difficult computing problems in multi-phase commit, distributed computing, the CAP theorem, distributed state, parallel code execution, and many others.

The ideal orchestration system should handle this. Let me emphasize that it is very very unlikely that there is an orchestration system on the market that can handle all of that. The maturity of distributed systems, from Kafka, Zookeeper, Cassandra, etcd, Kubernetes, Hadoop, and all of those are still in their infancy in properly handling these operations.

Nevertheless, any orchestration system of maturity has features (they are probably broken for dozens of edge cases) that attempt to help users perform them.

Execution: Do Not Despair!


Readers at this point are probably despairing saying “just write custom code in a programming language”.

But we’ve already gotten a glimpse of the summit. Orchestrators SHOULD be able to do these things. I’ve seen a half-dozen workflow editors and the host of different Vizio boxes demonstrate that these thing should be doable in the (somewhat loosely defined) scope of what an orchestrator can orchestrate.

While parts of an orchestrator that handles some or all of these very difficult aspects of the process/workflow WILL require dropping down to full-power Turing complete programming languages, an orchestrator of good design should support this ability!

Perl’s philosophy of “more than one way to do it” will probably be needed here, especially given how orchestration has so many nebulous origins and different terms for “processing”.

Integration

A hot term from the years of message queues, n-tier architectures, SOAP/XML web services, and still applicable in the microservices trends, the ability to integrate disparate systems and pools of data are critical to an orchestration system, which often doubles as the integration layer funneling data between heterogenous systems.

Mature orchestration systems often can support a great deal of interfaces out of the box, but that support is lost each time orchestration is reinvented and often requires adaption over time or more layers of wrappers (often in front of other layers of wrappers written for previous integrations or point-to-point interfaces) that degrade performance.

Mature integration is also leveraged by software vendors for lock-in. Because the core of these integrations is often closed source, even in the open source orchestration frameworks, it becomes difficult to migration orchestration to improved forms.

User Interfaces

A critical differentiator between good orchestration software and a collection of motley scripts, programs, frameworks and other program/tool hodgepodges is the visualization that (hopefully) well-designed user interfaces provide.

These can provide facilities for visualizing the defined processes and adapting them (editing of workflows), as well as the state of their current executions, scheduling, triggering adhoc executions, visualization of failure states and means to reattempt/resume execution, debugging, history, and metrics.

In particular, the UI for showing state machines / DAGs / workflows / Visio-esque diagrams is not trivial, and often isn’t implemented until medium level maturity of orchestration systems. Attempts to standardize things via XML standards have generally failed due to typical subterfuge using embrace-extend incompatibilities by vendors, and the XML standards have been fairly use case centric. A “generally solved” framework or design for this would go a long way to helping the evolution of orchestration without the cycle of perpetual full UI rewrites.

The UI rewrite subproblem is of course related to the Recurring Tragedy of UI Frameworks constantly being reinvented in every programming language, OS, and more recently, every new HTML/JS framework, but that is another rant. At least with HTML/JS we have a universal rendered and a long lived base language of HTML/CSS and (sob) Javascript.

These host of valuable capabilities are generally lost with each successive reinvention until late in the maturity of the implementation. A lack of UI with good historical continuity that other applications have like spreadsheets, word processors, databases front ends, etc is a huge loss for the industry and practice.

A way forward?

Obviously the problem is extraordinarily difficult. The balkanization of ?hundreds? ?thousands? distinct software that have “semisolved” this problem, almost all overpromising/bragging they can do things of this type but not reporting the fine print (or even aware of the full scope of the problem) abound.

I think this will only begin to be solved in IDEs, another Tragic Tale of Reinvention (Oh do I weep for your passing Turbo Pascal, Delphi, and your ilk). But IDEs recently have started to coalesce and stay more persistent. JetBrains IDEs have shown real staying power, and Visual Studio is now fairly open and has lots of language support. And Eclipse… is still actively updated too. Many of these IDEs or dialects of them have the basic tools (workflow display and editing, etc) to do this.

Fundamental orchestration visualization tools in these IDEs, from project structuring, code organization, and debugging/testing visualization, kind of like what gdb and compilers provide, could provide the basis of structure to apply all the way up the food chain of tools. And IDEs almost always can be limited to function as purposeful tools with limited plugins and options.

Configuration inheritance and how it locks you in

So a quick definition of "configuration inheritance". It's not a new technique or recent revolution. It appears practically everywhere since the olden days. Basically, configuration inheritance is a base configuration/environment/set of settings that can be overridden by more specific cases/situations/users/etc.

So you start with, say this base environment, we'll say it's .rootrc:

   USERNAME=system
   MACHINE_NAME=some_server
   HOME_DIR=/home/system

But then you actually are running as user DUDE, and they inherit the base system's environment and decorate some other settings, in /home/DUDE/.bashrc file:

  USERNAME=DUDE
  HOME_DIR=/home/DUDE
  BACKUP_LOC=/mnt/backups/DUDE
 
 But then DUDE uses github, which applies and adds it's own settings from the .github file:

  USERNAME=dudeman
  GITPASSWORD=githubpassword
 
This example is just inside some UNIX environment, highlighting that this has existed since the late 1970s, and probably before then. It shows up all over programming, software, and systems management. To show you a more extreme example, here is the Spring settings hierarchy:

Spring Boot uses a very particular PropertySource order that is designed to allow sensible overriding of values. Properties are considered in the following order:

1. Devtools global settings properties on your home directory (~/.spring-boot-devtools.properties when devtools is active).
2. @TestPropertySource annotations on your tests.
3. properties attribute on your tests. Available on @SpringBootTest and the test annotations for testing a particular slice of your application.
4. Command line arguments.
5. Properties from SPRING_APPLICATION_JSON (inline JSON embedded in an environment variable or system property).
6. ServletConfig init parameters.
7. ServletContext init parameters.
8. JNDI attributes from java:comp/env.
9. Java System properties (System.getProperties()).
10. OS environment variables.
11. A RandomValuePropertySource that has properties only in random.*.
12. Profile-specific application properties outside of your packaged jar (application-{profile}.properties and YAML variants).
13. Profile-specific application properties packaged inside your jar (application-{profile}.properties and YAML variants).
14. Application properties outside of your packaged jar (application.properties and YAML variants).
15. Application properties packaged inside your jar (application.properties and YAML variants).
16. @PropertySource annotations on your @Configuration classes.
17. Default properties (specified by setting SpringApplication.setDefaultProperties).

... I can hear Harry Caray going "holy cow". That hierarchy has lots of sensible features that involve testing, dev vs stage vs production environment settings, ability to do adhoc overrides, defaults, etc. There's reasonable justification for such a complicated stack, including evolution over time and hard learned lessons.

Really what this data pattern is a series of maps / dictionaries / key-value files that are then stacked upon each other, and the lookup then starts at the top of the stack and descends until it finds a value of a given key.

You see it in the above examples, in kubernetes/docker image overlays, in practically every infrastructure management tool, and dozens of other places.

Why? this is a good tool for centralizing common properties at one level, and overriding in specific situations, in order to manage settings across a variety of environments, or machines, or situations, or use cases.

It also enables a key aspect of software development and systems management: iterative/evolutionary development, as new requirements, systems, interfaces are added to something, or in the case of pure software development, successive releases to support increasing complexity.

But what seems like a solid foundation really is a teetering tower once you get five layers of configuration inheritance. What provides the illusion of management simplicity completely falls apart when significant reconfiguration occurs at the lower levels of the "stack of maps". And that is a key way in how organizations are locked into systems and software packages.

Here's the key feature that almost every one of these configuration inheritance / stacked maps lacks: a basic meta-mapping feature that explains where each value comes from. Using our first example of unix environment variables, which are displayed in any "context" by using the env command:

    USERNAME=dudeman
    GITPASSWORD=githubpassword
    HOME_DIR=/home/DUDE
    MACHINE_NAME=some_server
    BACKUP_LOC=/mnt/backups/DUDE

So my suggestion is that a key facility should be the metaenv command:

    .github::USERNAME=dudeman
    .github::GITPASSWORD=githubpassword
    .bashrc::HOME_DIR=/home/DUDE
    .rootrc::MACHINE_NAME=some_server
    .bashrc::BACKUP_LOC=/mnt/backups/DUDE

This command will show the additional information about which of the three files (in this example) actual values were assigned or possibly overridden, so that if there is a problem, bug, or unexpected value, someone can walk up the chain.

Unfortunately, once you get to the concept of multiple users, applications, servers, environments, etc etc, you form a huge tree of different inheriting situations, much like the huge inheritance hierarchies of OOP software and other situations.

The real problem with that is that trees are very hard to visualize with tools, especailly command line tools. But they are possible, and that is another class of tool that also does not accompany these systems, but should.

Because, should you apply a change to a file that affects a lot of other files that inherit from it, it would be nice to visualize all the impacted "downstream" files, and ALSO which downstream files are NOT affected because they override the values you changed.

NextStep OS fits in processor cache.

Back in the day of me buying a 486 PC for my college days, I remember thumbing through Computer Shopper. Somewhere in those days someone took the cache memory for processors and made a whole computer memory out of them. It was a lot more expensive!

But since those days and with the massive growth in silicon for L2/L3 caches of modern processors, I reflect on those days of operating systems and wonder "what if we could fit our OS and its apps in cache"?

Old Windows 3.1 or MS-DOS was not a pleasant experience compared to modern operating systems, almost no one can deny that. But NeXTSTEP? The color variants that came out after NeXT gave up on its black and white cubes... that is basically a modern UI, and arguably better than modern Linux GUIs.

https://www.youtube.com/watch?v=TIrTh80Z8jw

Old NeXT cubes ran on 16-64 megabytes of RAM and 25-33 megahertz of CPU, and 400 MB to 1 GB of disk. I can't find good numbers on the color variant that ran on x86, but I think our machines at college ran that very nicely on about 32 megabytes of RAM.

The prime curve of Moore's Law has produced quite the hardware bounty. Old timers like me bemoan a seemingly bloated software layer that has grown over the decades. The current Ryzen processors boast 80 megabytes of cache, and of course RAM is dollars a gigabyte, and SSD storage is getting reasonable.

Holy crap, NeXT would run entirely in cache. OK, swap might start getting into RAM. Modern RAM comes in the tens of gigabytes, much bigger than the hard disk on those machines. It basically looks like a modern OS, with a bit less aliasing on the fonts and icons. Productivity app set is practically identical. It had TCPIP networking, an HTML 1.0 web browser, spreadsheets, mail, etc.

With the end of big leaps from Moore's Law, if we want more performance, we'll need to optimize the software stack. I think a really cool project would be to start with the basis of NeXTStEP circa 1995-1997 and very carefully add in only what is needed. I remember the Mach kernel being buggy, but that can be swapped out.

So many "what is a microservice" posts...

First off: I am not a zealotous proponent of maximum microservices. There are problems galore in constantly changing services that are also interdependent on each other. But the sheer number of bad microservice posts trying to detail what they are and why they have cache is astounding. It isn't that hard, people.

Most of these bad introductions starts at the software/component/design level, trying to compare architectural diagrams or styles, and contrasting with "monolith" architectures. ...Totally misses the main point of microservices that drives their benefits and values. Microservices fundamentally don't

Microservices are small in the sense that they are easily integrated atomic unit into modern cloud/devops/infrastructure-as-code/DatacenterOS computing and continuous delivery platforms. That is a microservice: whether it's a small VM, a docker image, jails/LXC, whatever. They are easily provisioned, deployed, relocated, and redeployed on these platforms.

Microservices plug into continuous delivery and version control systems with automated testing and immediate deployment to at least qa/staging environments once the build passes (and rapid deployment to production once the tests pass). Because they are "small", the incremental builds are relatively quick (an hour or two tops) even though they use lots of modern code coverage, code quality, code formatting, automated testing, automated deployment, and similar tasks beyond simple compilation.

Here are things that start to make your code not-microservice:

- proprietary software: There's a reason microservices are built to the wazoo on open source software. License servers, license acquisition, audits... always introduce a screeching halt to devops. Sure that can be bolted on, but there often are even more insidious things that get in the way: poor command line tooling, over-reliance on UI configuration, hidden "consultant bombs" and obfuscated documentation that prevent you from efficiently making a continuous integration pipeline or versioning your code with it.

Part of the small code and huge numbers of servers in microservices actively defends against the insidious influence of proprietary software salespeople: Sure organizations will buy 10 or even 100 licenses. But 10,000 Websphere licenses? Forget it. Microservices by preempting proprietary software enables scaling down the line. A definite feature in most circumstances.

- manual deployments: I get you might need a production approval process in many situations. But if each deployment requires some ticket to be filed and fingers drummed on a desk while you wait for some long list of manual tasks to be performed... forget about it.

- hardware starvation: while hardware costs shouldn't be ignored, if your organization can't deploy a $5,000 server without $50,000 of meetings, approvals, and documentation, or if your "cloud" is constantly running out of servers and disk space to power your software, then you can't do microservices. The devops platforms it depends on require sensibly budgeted hardware that can handle new services. That doesn't mean a small piddly service needs a quad-core processor and 4 gigs of ram. It means that the autodeployment step in the continuous build only fails due to hardware shortage once every few months, and that is fixed within a day, tops.

If your huge 1,000,000 line "monolith" legacy service can be fit into the above: containerized, continuously delivered, and deployed within a few hours of pull request or some similar software change, then congratulations: you are a elite microservice team.

- multiple organizations served in a codebase: Per Conway's law, your software WILL align with your business organization. Some of the problems of the big monolithic ERP software was to couple disparate business divisions in the same system. So programmers maintaining software couldn't find the parts of the monolith they needed when they were inevitably aligned with some business function. Microservices help keep enterprises' custom software isolated in specific functions that can be documented and maintained more easily. If some function is to be reused (but inevitably... slightly... different) across organizations, microservices should implement the shared model, and then write dependent services for the specific organization variants.

Cloud and Devops: eliminating an entire layer of assholes

I've been in quite a few gigantic enterprises. Medical, Insurance, Retail, Government, you name it.

Without exception, in every one of these organizations the infrastructure teams function as blights upon innovation and adaptability, and impose unreasonable costs and delays. Infrastructure is not simple, I will grant... and this is not revelatory news to anyone in the industry.

It used to be, oh, say, a decade ago that we heard only rumors of Google, Facebook, Amazon, and other firms with dynamic, adaptive, cost-effective, anticipatory, and plentiful infrastructure for testing, prototyping, and production. No nightmares of 8 extra GB of RAM costing $1,000, or a simple PC server carrying $10,000 / year in upkeep chargebacks.

Once rumors, AWS / Rackspace / Google Compute and a host of other cloud providers have made that a much more stark contrast to anyone mired in internal infrastructure. Once things like Chef and other automated provisioning come into play, the contrast becomes even more pronounced.

Again, none of this is revelatory. Clouds are better cost, more adaptive, efficient, allow you to discover new technologies and make those practical to your job, group, and enterprise. Anyone who has done a cloud-enabled project has experienced the nirvana... But only now when we are being cast back into the rats nest of mandated internal infrastructure use do I recall all those meetings and emails and total clusterfucks:

• your RAM is too big, we can't support more than 8GB. Oh, you want 4GB? That will be $1000/year extra
• oh, you need another 100 MB of disk space, that will be $1000/year. No we won't provision you extra space, you have to crash in production and then raise a ticket to get us to do anything.
• Let's schedule 10 meetings to discuss minor hardware spec increases that should be 1/100 of the labor waste of the meetings
• Hardware guys bloviate about what your application's hardware spec requirements are, even though they have NO IDEA about what it does or its user base
• CPU? You like 386s right? 
• Oh, your VM is colocated with other extremely high use VMs? Too bad, so sad.
• SSD? What is that?
• Cloud? We have an internal cloud. No we don't have elastic storage or API provisioning. We can get your new VM up in 5 months rather than 1 year!
• Accounting... ah, the accounting. "It's free... if you can get it". "It has no upkeep cost, but it costs a ton to setup". "Will the cost drop over time as Moore's Law marches on? NO". 

I am so fucking sick of it. Thus I DECLARE:

The #1 Feature of Clouds is the Elimination of an Entire Layer of Assholes From Your Life

That's right. #1 feature by far. No one from our cloud provider has ever sent us threatening emails yanking our RAM size, or down-specced our machines without consultation, or dragged their feet to provide new machines, disk space, CPU, whatever. Or spent weeks ignoring tickets behind ticket walls. No one scheduling bloviating meetings and handing down arbitrary new policies from above. No meddling in your application architecture. No complaints to the CEO if you decide to use someone else's infrastructure. None of it.

Not just any arbitrary layer of assholes. Infrastructure lends itself to controlling and sociopathic shitheads. They get to ooh and aah over high-spec machines, they get large lump-sum budget line items, and, best of all, they get to lord over the distribution of those resources to the people that need them, and demand asskissing and tribute. This is a particularly obnoxious layer of entitled, intractable, petulant assholes you are extricating from your life.

What is the #1 downside of our collective profession? Stress. Stress that causes measurable, documented deleterious health effects. If you can identify something that significantly reduces that stress, and I argue clouds can do that, then it should be a priority: personally, professionally, and industry-wide.

Language Evolution Battle Royale.

So I read a lot of programming blogs and the inevitable debate about languages. Rust, Go, Java, Groovy, Ruby, Python, Javascript, C, C++, CSharp, etc.

What would be really fun is a long, thoughtful, crowdsourced, dynamic converging evolutionary practical language bracket shootout.

You start with a March Madness style of bracket. Not random, but the MAJOR languages seeded by their popularity.

Lesser seeds go to languages of note, usually in-use languages of lesser popularity with nonetheless some valid use case, but also emergent languages (Go, Rust), and some important historical ones not yet reproduced (Ada, APL, Smalltalk, ?Delphi?).

Lisp doesn't get in. If you've ever dealt with a lisper (how would you know... DON'T WORRY THEY'LL TELL YOU) you know they just aren't constructive members of any practical language discussion. Haskell is probably also not in for basically the same reasons.

For each language in an individual match, the features, standard library, runtimes, and debated. The outcome of each match is an evolved merged language... one that takes an effective compromise of the two languages, making sure to keep in mind the seeding of the sourced languages: if one language is bigger seeded, the reasons for its popularity/dominance can't be dramatically undermined in the evolutionary product.

Each evolutionary advancement in the bracket should note what was left out.

I (at the bottom of the blog, assuming bracket ninja stays around longer than next Tuesday) basically structured 16 languages. I only seeded the top four: Java and C++ are number one seeds, Javascript and Python are the #2 seeds.

From there, I didn't go by the usual ranking. Instead, the "competitors" for the immediate opponent and hopefully the second-round opponent deal with the primary domains of the language.

So C++ is obviously from the C family of languages and exists primarily as a systems language and, due to popularity and historical gravitas, is in many other domains. While Objective C isn't really a systems language, it is targetting devices with some aspects of performance and embedded problems. It also is a direct descendent of C as well. Objective C has strong recent popularity in TIOBE, but this is mostly due to the authoritarian dictates of Apple rather than a matter of public choice. Anyway, since these are both close in lineage and similarity, lets have them blast it out in the first round to produce a reasonable direct-from-C immediate evolutionary step

Similarly, Java is paired with CSharp. Both are essentially enterprise programming platforms, and quite similar in approach. So let's have them battle it out and produce the evolutionary result of managed portable C-esque languages with great standard runtimes and deep software ecosystems.

Python and Ruby are paired in the second-seed battle. Both are high-productivity scripting languages intended for both small and medium scale software packages, and dabble with aspirations of large-scale code. Both have their primary mission for fast-start web applications and the leading scripting languages. Both have worked hard to leave their scripting limitations behind with LLVM runtimes for performance and modern multithreading abilities.

The last seeded battle is Javascript and Dart. Javascript is seeing massive adoption rates due to the web (duh) and has made notable forays into other areas (Node.js, etc). Dart is the most obscure language on here with the least adoption, but is the most preeminent challenger to address JS's massive shortcomings as a fundamental language. Here also will likely be the battle to determine the fate of prototyping as a viable OOP feature moving forward.

It should be noted that most of the four top seed battles are really to address the glaring issues in the main four platforms.

The other four faceoffs all relate to the set-of-four subbracket they belong to.

 So Go and Rust are addressed at systems programming like C++, but much more exotic. The second round mashup between Objective++ and GRusto should produce a systems language par excellence.

Scala and Ada are two enterprise-focused languages (Ada has strong historical strength, once the #3 language in TIOBE). The eventual hybrid of Java# and Scalada should produce a featureful yet very performant and productive enterprise programming platform with strong typing and scalability.

Visual Basic and Smalltalk are two high-productivity languages with strong IDE designer components and RAD aspects, aligning with Python and Ruby high-productivity loose-typing applications. Smalltalk's legendary status as the ultimate IDE-supported OOP language needs to be represented. Visual Basic, despite the guffaws, has produced more applications in its programming for the masses than most non-top-seeded languages. The outcome of BasicTalk and Rubython should produce an amazing high-productivity scripting application building capability with phenomenal database integration and UI construction from desktop to web.

Finally, there is Perl and PHP. PHP is too popular to not include. Perl is too popular to not include. Both are viewed as intermediate steps in the evolution of their domains, and remain very heavily used. PHPerl should cull whatever is good about these platforms not that was missed by their successors, be it simplicity, frameworks, syntactic sugar. Combined with JavaDart in the second round will produce a useful, flexible language with unique programming constructs and a powerful standard library with nods to other domains such that it can expand into other places. This overall bracket represents the "ugly but effective" group of languages and platforms that produces an enormous, undeniably proportion of modern software and can't be ignored.

From those four domains, combined evolutions may be harder.

What is left out? Well, the free bracket only allowed me 16. I think that Ruby/Perl should be subjected to a play-in game before facing Python, or strangely Python/Perl face off. Ada should probably be subjected to a play-in bracket of many other languages in general systems use from the ages.

The lack of R/APL/Fortran and other heavyweight array/matrix/stats/math processing should probably be inserted into the systems sub-bracket. Go and Rust can battle it out in a play-in prelim, and then incorporated with the powers of the raw computing languages.

Scala should be representative of a grand JVM language battle between groovy, scala, and all the others.

In the end, we see if some comprehensive language can be ideated that isn't the tragedy of the commons. 

Stoopid Idea of the Day: Five lines or less

Oh hey, an interesting StackOverflow question:

http://stackoverflow.com/questions/811074/what-is-the-coolest-thing-you-can-do-in-10-lines-of-simple-code-help-me-inspir

Of course it was closed as unconstructive.

How about five lines? One hand, one statement per finger, if you need that.

Pointer finger: demonstrably state the input instruction/line

Middle / Ring / Pinky: demonstrably process the input

Thumb: demonstrably output or return (ok, if you really need to do some processing here in the return...)

Line count doesn't include basic stuff like function bodies, class overhead, imports, etc. 

Also, state language, environment/os/libraries needed.

If your "line" is longer than, oh, 120 characters, then is that really a line? As in, if you are a master of obfuscated C/C++, please, just don't...

APL is also cheating. 

Heavy UNIX commands... well, I guess everything is relative...

People: readable five lines of code that perform a useful, adaptable, extendable purpose.

Some good tagging and relations plus moderation... 

Yet another build system? Please?

Problem: Most gradle/ant builds turn into groovy/XML soup

Problem: Gradle/Ant projects can lack a good idea of "where to start" to build something. What properties do I need to define? What order of tasks? What do the tasks depend on (db, etc)?

Problem: Maven is a PITA and inflexible, and the advocates are tyrannical

Problem: Maven still ends up with a massive huge xml file

Problem: Maven won't do simple, literal directory-of-jars

Problem: Maven's internet connection dependence

Problem: If maven dep-resolve fails you (IBM Websphere, another reason to hate you), you're basically screwed.

------------------------

What are the good parts that have been invented:

- (Ant) a general set of plugins for templating/replacement, files, building code, etc etc.

- (Ant) decent IDE support

- (Maven) maven repository: the failed dependency resolution is a rare case. Generally, this thing is a very good thing for code sharing and library use/reuse. Gotta leverage this massive body of work and mindshare.

- (Maven) standard project structure: although src/main/java isn't my favorite thing (three levels down? really?), the overall maven project structure provides acceptable structure that people are used to now.

- (Maven) standard project structure for WARs: again, I am not too excited about the file structure, but it'll do.

- (Maven) standard jar naming <artifact>-<version>.jar, a great thing indeed

- (Maven) binary / javadoc / source jars

- (Gradle) groovy scripting for those things you need scripting

-----------------------

What I want from a build system

-  leverage the structuring standards of Maven that have made unfamiliar projects more straightforward to interpret

- leverage Maven repos and dependency maps, but add flexibility fallbacks for literal jars, and only do dependency resolution when new projects are setup or new libraries/dependencies added (one-off or one-time tasks)

- this is a new feature for java builds: a single file download that specifies where to download the rest of the project (sourcecode, etc). The build system will connect to sftp/ftp/git/subversion/whatever to get the code, then do dep resolve. Kind of like .rpms for linux. Great way to do starting points for projects, especially ones without a fancy site or github...

- use some of the nice features that make Ruby on Rails so good: convention over configuration, and folder structure as configuration. Maven failed at this since you end up with a huge POM xml. 

  - Why not have a naming convention for mulitple source folders: as long as they match /src-* and are at the project root, then they are an additional source folder. 

  - /info dir has the project info. 

  - /lib for various libraries. In the /lib, you can have .jar files for inline/literal inclusion, or an empty file named <artifact>;<version> that triggers dependency resolution. Nice instant  visual indication of the libraries used to build a project. Subdirs for main for main build, test for test build and execute, etc.

  - /tasks is a dir with more subdirs <taskname>, and each of those has their own gradle/ant script that can be invoked

Need to add a quick jar? drop it in the /lib/main dir. Need to add a quick set of dependencies from another project that you know works? Copy-paste the dependency files into the lib dir. Source tree getting a bit thick in the /src/main/java folder? spin off some more src-* source directories. Maybe even tasks can become drag-and-drop or copy/paste with sufficient automation/decoupling. 

Web resources, ears, subprojects, project relationships, etc: all to be determined. 

Another key aspect should be IDE support and multiple IDE usage. perhaps a separate directory tree for ide-specific settings. Ant/Gradle/Maven xmls can be autogenerated for IDE import/usage if need be. 

Project structure:

/	root of project
/lib	libraries and dependencies
/lib/main	libs and deps for main compile
/lib/test	libs and deps for unit test exec
/src	maven-style primary source dir, with /main/java, /tast/java, /main/resource, etc
/src-	more maven-style source dirs, such as a service impl, etc
/task	nonstandard task scripts

Frameworkitis and False Enterprisey Security: WS-Security and signing.

http://blogs.msdn.com/b/vbertocci/archive/2005/04/25/end-to-end-security-or-why-you-shouldn-t-drive-your-motorcycle-naked.aspx

Who wears a motorcycle helmet indoors? Just because it's a simple analogy doesn't mean it's right.

This is still rampant frameworkitis. HTTPS isn't good enough! What happens if something is done after the message reaches it's point?

Oh, you mean the fucking ENDpoint?

99% of SOAP bloat systems (aka, licensed application servers configured and installed by consultants) IMMEDIATELY deserialize a SOAP message to a datastructure, and from there that data structure will be mangled by database call or MQ message or whatever.

Notice the uselessness of message signing. It's a false security blanket sold to stupid fucking executives. The data will be plaintext in a host of different settings post-parse: SQL requests, followup messages. That requires either good internal data practices or specifically tailored security architecture.

READ: less than 1% of cases, and outside the concerns of SOAP, which is a messaging protocol, not a transport protocol.

It's frameworkitis and enterprisy. Let's add a bunch of stuff that isn't the concern, bloat, bloat, bloat, million dollar license, buy a ferrari.

Fucking software companies. 

A little Spring XML to JSON conversion experiment...

What might some random Spring config (in this case a Spring-batch example) look like if JSON were the configuration/injection message format? What is good, what is bad?

 <bean id="taskletJob" parent="simpleJob">  
   <property name="steps">  
     <list>  
       <bean id="deleteFilesInDir" parent="taskletStep">  
         <property name="tasklet">  
           <bean  
       class="org.springframework.batch.sample.tasklet.FileDeletingTasklet">  
             <property name="directoryResource"  
               ref="directory" />  
           </bean>  
         </property>  
       </bean>  
       <bean id="executeSystemCommand" parent="taskletStep">  
         <property name="tasklet">  
           <bean  
       class="org.springframework.batch.sample.common.SystemCommandTasklet">  
             <property name="command" value="echo hello" />  
             <!-- 5 second timeout for the command to complete -->  
             <property name="timeout" value="5000" />  
           </bean>  
         </property>  
       </bean>  
     </list>  
   </property>  
 </bean>  
 <bean id="directory"  
   class="org.springframework.core.io.FileSystemResource">  
   <constructor-arg value="target/test-outputs/test-dir" />  
 </bean>  

My swing at JSON, using JSON-ish features and advantages (lists, maps/objects, etc)

 {  
   "taskletjob":  
     {"#t":"BEAN", "id":"taskletJob", "parent": "simpleJob",  
     "properties": {   
       "steps": [  
         {"#t": "BEAN",  
          "id":"deleteFilesInDir",   
          "parent":"taskletStep",  
          "properties":  
           {"tasklet":  
             {"#t":"BEAN", "class":"org.springframework.batch.sample.tasklet.FileDeletingTasklet",  
              "properties": {"directoryResource" : {"#ref":"directory"}}  
             }  
           }  
         },  
         {"#t": "BEAN",  
          "id":"executeSystemCommand",   
          "parent":"taskletStep",  
          "properties":  
           {"tasklet":  
             {"#t":"BEAN", "class":"org.springframework.batch.sample.common.SystemCommandTasklet",  
              "properties": { "command": "echo hello", "timeout": 5000}  
             }  
           }  
         }  
        ]  
       }  
     },  
   "directory": {  
     "#t":"BEAN", "id":"directory",   
     "class":"org.springframework.core.io.FileSystemResource",   
     "constructor-args": ["target/test-outputs/test-dir"]  
   }    
 }  

So what I found out:

Size in LOC was mostly identical. No real big savings there.
Spring utilizes namespaces implicitly or explicitly... No real JSON mechanism, so I invented the "#t" which is supposed to indicate the object type if it is typed, to give guidance to some parser/bean construction code as to required fields, etc.

In other places, like properties, the JSON object is used more as a map, since it is more intuitive that way. The JSON format here seems much more natural, and seem to save lots of keystrokes. I mean would you rather type:


<property name="command" value="echo hello" />
<property name="timeout" value="5000" />

or

"properties": { "command": "echo hello", "timeout": 5000}

Clear win for the JSON there.

However, JSON has some weird corner cases in formatting, such as arrays of objects. How do you render that without wasting extra tabs or lines? Granted, the <list> construct in spring xml wastes a tab and a line...

And all the double-quotes in JSON. Annoying. Not as annoying as xml ending tags. And why are commas even needed?

Other JSON problems: Getting lost in curly-brace land. WHile annoying verbose, at least closing tags in XML let you see what's being ended/closed. But considering most modern editors have curly brace features for C and Java code, this isn't a huge deal.

But if I had my druthers with a variation on JSON, where you don't need double-quotes on the object keys, values are optional, and drop the commas, why can't JSON be this?

 {  
   taskletjob:  
     {#BEAN id:"taskletJob"   
     parent: "simpleJob"  
     properties: {   
       steps: [  
         {#BEAN" id:"deleteFilesInDir"  
          parent:"taskletStep"  
          properties  
           {tasklet:  
             {#BEAN class:"org.springframework.batch.sample.tasklet.FileDeletingTasklet"  
              properties: {"directoryResource" : {"#ref":"directory"}}  
             }  
           }  
         }  
         {#BEAN id:"executeSystemCommand"  
          parent:"taskletStep"  
          properties:  
           {tasklet:  
             {#BEAN "class":"org.springframework.batch.sample.common.SystemCommandTasklet"  
              properties: { command: "echo hello" timeout: 5000}  
             }  
           }  
         }  
        ]  
       }  
     }  
   directory: {  
     #BEAN id:"directory" class:"org.springframework.core.io.FileSystemResource"  
     constructor-args: ["target/test-outputs/test-dir"]  
   }    
 }  

... Not a huge difference, but it would make reorganizing and moving object chunks around without worrying about breaking the commas in a list, or the fields inside an object. I think the parser would be about the same, no impacts to speed. But JSON the format spec, that ship has sailed... Maybe it looks too much like S-expr...

Oracle WINS!!!!.... what?

So Oracle has won... something.

I'm scared that Google has won this with a spectacular Oracle pyrrhic victory. But we as software developers may be losing in a big way.

If Oracle gets to copyright a basic rangecheck function, things are going to get really annoying.

I wish I could talk to a copyright judge and make this argument:

"Can you copyright the word 'is'?" (I assume no)

"Can you copyright the words 'greater than'?" (again I assume no)

"Can you copyright the words 'greater than this but less than that'?" (again I assume no)

And really, right there, you have pseudocode for rangecheck. For the java version, well, basically, you're just with a dialect translations: "Can you copyright the words 'superieure a cette mais inferieure a celle'?" (French google translate, and again I assume no).

I mean, it's not even a quote. There is no depth of meaning, or abstract reference, or hidden commentary. It is a direct request statement, basic functional language. How can that be copyrighted?

I mean, imagine the lawsuits that come out of this. Patent armories don't help for defense. Only writing tons and tons of library code and copyrighting it. You might have tons of excess software written just so they can sue other people for doing similar stuff. 

Windows 7 File Explorer and Search: WTF?

This begins to make me question the fundamental information freedom of the internet and the astroturfing ability of major corporations.

Windows 7 gets sooooo much love. Okay, you finally cleaned up the kernel. Well, we'll see. At work I still have a dozen security nanny applications. But the desktop. What the hell? People say Microsoft has a great research division and tons of usability people. WHERE ARE THEY?

The Windows 7 tendency to hide the Menu in every application. WHAT THE HELL? Why? Why? Whhyyyy? You're relying on people to know to press ALT to get to the menu? Does microsoft not want people to use anything except toolbars in applications? Toolbars rely in intuitive icon design, and intuitive icon design hasn't even left the stone age. 40 years of GUIs and 20 years of the web, and icons still are incomprehensible until maybe, possibly, once you've clicked on the magic button to see what it does, only then may you understand what an open door symbol actually means in some random application. You need the menu, you idiots.

The main toolbar yanked out so many features from the XP toolbar, it isn't even funny. Not having a folder-based organization capability for nesting your start menu links? Am I the only one that had dozens of environment-specific and project-specific links for SSH, html pages, documentation, etc that I would organize in the start menu for two-click access? I'm stuck dragging some folder called "Launch" to the toolbar, and since they wont allow you to drag around the order, guess what? I have to do:

000-link1

010-link2

011-link3

020-link4

Old School BASIC programmers should be nodding their heads right now.

The search functions. I used advanced search and find-in-files a lot. A LOT. Where are they? Gone? And when I search for a file that's in a bunch of deep folders, it gives me four results... and no way to resize the columns so I can see what folder they are in:

That is so bad. Soooooooooooooo bad. Basically, Microsoft is begging everyone to install Google Desktop Search. Oh yeah, and trust us with Bing, we really know how to make search work. See how well we did it on the desktop? Idiots.

Other random crap on the Explorer: As you navigate down the folder tree on the right pane, it doesn't auto-expand the tree in the left pane? WHY NOT? I will give them some cudos for the breadcrumb path. That's not bad. And getting rid of most of the "My " folders and just doing , such as Pictures Videos Music, etc. Hey, it only took you 10 fucking years to stop doing that. Morons. They still do ProgramFiles though. Facepalm.

It's like they wanted to do half of what Nautilus does on Gnome, but got in a huge argument and just stopped finishing it. Basically, it's like a cake being cooked by divorcing parents that spent the entire time making it screaming at each other, not even noticing it was overcooked, they misspelled your name, and didn't put the right number of candles.

Microsoft continues to not care about their customers, and then play mystified, vulnerable child when people call them on their decades-old bullshit. When your company is basically symbolized by a narcissistic, infantile, manipulative, spoiled brat for 20 years, you've got a problem.

I suppose I shouldn't have been surprised.

Lisp: Show me the Beef.

This is the wendy's moment.

God do people like talking about lisp. How amazing and transcendent it is.

Now, I tried lisp. For one semester of Scheme in my programming languages class, unfortunately taught by a teacher who was unable to communicate to humans, and not because he was foreign.

I never went back. Don't even think about going back. Not even a little. Of course I bombed out of the Math department, and most lisp people are math people.

Still, lisp is beyond hype. It is the most mythical and legendary thing in computers, which is bizarre since usually the subject of such things has passed in antiquity. Lisp is right there. Go ahead, there's a dozen implementations. Even a JVM one. The legendary power, right there.

Here's the catch. And this applies to haskell too: WHERE ARE THE APPS? I've heard ludicrous stories of lisp hackers bragging 100x productivity over other languages.

Okay.

Where are the apps?

C begat the UNIX operating system, DOOM, databases, and many office suites. That is a serious software resume.

What has lisp given us? Emacs? Anything else? Anything?

Java doesn't have as amazing a resume as C does, but it does run huge numbers of web sites. Those are killer apps in their own right, transforming your life. How many sites are written in Lisp? That should be some low-hanging fruit. Nope, ruby beat you to that.

What is apparent after decades, let me repeat, _decades_ of existence, through the expansion of computing power by a factor of 1000, lisp has not delivered on its legend. Not even minimally.

Instead, what is apparent is that it is a cult language. This comparison is so appropriate, since the cult of lisp subdivides into even more subcults, even down to the individual. Between the non-infix notation, parentheses, macros, individually produced programming paradigms and features, one man's lisp code is unreadable by another.

If lisp was the most freedom in programming possible, then it used its freedom to shoot itself.

I'll grant the lispers this: if you write lisp, your IQ and ability are measurably above other programmers. And that is the problem with lisp. For every story about the lone lisper beating an entire competing programming department's output in some circumstance, I can't distinguish between the lone genius producing a lot of software in any language/ecosystem from the language/ecosystem itself.

Everyone knows that a single smart programmer can often beat the output of 5-10 average people having to write specs, communicate, have status meetings, status reports, agree on interfaces, deal with different communication vocabularies, and different abilities.

But it just doesn't scale. Lisp doesn't scale. Lisp programmers don't scale. Lisp programs don't scale.

And that's why the crowning achievement of Lisp is a text editor.

Weeds, Flowers, and Fruit Trees

Oh, you have a new language that will revolutionize the world?

JVM languages are sprouting. It's getting a bit ridiculous. Fun question for someone with more time than me: are there now more than 100 JVM languages?

Please, grant me the power of a weed whacker. Almost all these new languages are nothing but weeds.

See, you have your staple crops in the garden: wheat, corn, potatoes, etc. This is what Java-the-language (and software ecosystem) is. Not flashy, but it has many compilers, a full runtime, massive IDE support, libraries, and accessible to programmers of many abilities. Just like developed crops are fairly easy to grow, yield well, and basically get used to make white bread. It has decompilers, debuggers, monitoring, tracing, and tons of other ecosystem features.

Then there are the weeds. Sure they may look different or do a few things better than Java, but for the most part they lack IDE tooling, decompilers, monitoring, lots of good examples. Basically, not worth your time, but even worse, they are taking up cropland, water, fertilizer, and other resources that better plants and plant ecosystems could produce.

There are flowers. Oh, these languages look pretty, do amazing things. But you can't eat them, or make textiles, or animal feed. They have all the minuses of the typical weed language, such as crappy IDE support and ecosystems, but you keep them around since you can pluck it occasionally for fun, or for some specific purpose.

Fruit trees would be the best, lots of food, persist year-to-year, solid foundation, but beautiful.

Not a lot of fruit trees.

Anyway, the bar needs to be raised. Your compiler and syntax description isn't enough. A new language isn't a new language until it has:

- compiler

- IDE tooling:

- syntax highlighting

- code completion (assuming it has sufficient static typing)

- syntax error dynamic detection

- line-by-line debugging

- runtime variable/stack/etc debugging during stepthrough

- clean integration with existing java libraries, even if you have your own runtime

- a good set of examples that demonstrate the feature set of the language

That's just the bar for announcing your practical arrival on the scene. That's not even the beginning of being somewhat mature:

- decompilers, well, if the language becomes common enough

- web frameworks, enterprise frameworks, GUI frameworks, etc.

- web services, templating, and a host of other libraries that are specific to your language's style

- unit test frameworks, build frameworks, etc etc etc

- lots of programmers

- I could go on a long time. Basically, look at Ruby's rise and all the things that were necessary for that to not be a toy language: webserver that didn't constantly crash, optimizing VM (JVM basically does that for you), web framework (Rails), templating frameworks, and lots of other things.

And you know what? That's not even good enough. How does your language change the craft of programming? You know what? Ruby didn't do squat for programming. Ruby on Rails changed the craft of programming. Ruby + Rails was a legitimate new language.

My fundamental bitch with new languages is that they fundamentally don't solve problems any better than Java. They don't make it faster, they don't really change most day-to-day programming. Most make programming more confusing to people. Even groovy basically requires you to know java programming before you can learn to groovy. It's IDE plugins still suck (well, under eclipse). I think the typical JSP plugin is more useful, and that's pretty damning.

Java itself is careening toward C++ hell. There is so much in C++, it might as well be impossible to learn all the features. Considering the compilers barely implement all the features at a bare minimum of support (but pray, don't expect, for good compile error reporting). The features are bolted on. Annotations have basically become C macros.

A lot of what made Java great had nothing to do with the language. The language was a C-like dialect engineered solely for being easily understood by the majority of programmers. Java's killer features: JVM, cross-platform, cross-platform runtime library, internet support and applets, namespacing, unicode support, garbage collection, and FREE.

Compared with C programs, you didn't need an unreadable soup of macros for "cross platform" adaptation, didn't need to worry about the actual size of ints, eliminated lots of memory leaks, eliminated a ton of security holes with non-pointer math strings. It was so much easier coding in java, even if the language was boring.

The new languages are really just shifting around chairs on the deck of the cruise ship JVM and it's optimizing runtime, garbage collection, security features, libraries. The JVM provided a

Most languages are just feature-loading more inheritance options (multiple inheritance, mixins) and doing syntactic sugar (closures, autoboxing, type inferring).

What we really need is new ecosystem features: packaging and dependency management (OSGi, ruby gems, linux packaging methods, maven, namespace version tagging), cross-language compiling (coffeescript), monitoring (JMX++), features for modularizing code so these huge dependent codebase package trees don't grow out of control, a new messaging infrastructure for XML/JSON/whatever with integrated parse/query/path lookup, url-based classpath (not filesystem paths, urls, with a local cache), maybe integrate a RESTful storage paradigm like sling and JCR/jackrabbit for code and configuration storage, better dynamic reloading of code support, fundamental support for varied sizes of devices (micro/super-low footprint, smartphone footprint, desktop/server footprint), good support for scaled execution, RMI, multithreading.

Get me a new ecosystem, not a new language.

I hate game theory questions

http://www.techinterview.org/post/526325766/pirates

I hate this question, at least it's statement. A lot of these game theory questions are bandied about, but they are loaded with unspoken assumptions about human behavior behaving in perfectly logical rigidity to unspoken assumptions.

The unspoken assumptions here are:

- the pirates will obey the rules (uh, they are pirates, why would they consistently obey) over time

- pirates will not bargain or collude, except in the way that is laid out in the explanation

The strange thing about this "puzzle" is that it assumes perfect adherence to proposals and rules, but assumes no one can communicate alternatives.

If the result (98 - 0 - 1 - 0 -1) is so inevitable, then why wouldn't the pirates (0 - 1 - 0 -1) countercollude to kill the leader in exchange for equal distribution, under the express conditions of giving up the right of decision of division based on rank?

What we have here are five agents. The prioritization only matters for the first case if the other four agents (actually, three can collude, leave the least popular one out in the cold, and kill the top man). If the deconstruction is that the initial collusion leads to the four situation, then the remaining three can collude to drop it to equal division amongst them and kill the top man, and so on. Since the number one priority (unstated) is living over being rich, this collusion strategy would force the divider to think of his future and be more equitous.. Under this collusion strategy, the first leader to not be greedy lives, and keeps money, with security.

In my opinion, the outlined strategy would only provide a tentative peace until the others determined a collusion strategy. The fact that there exist deconstructions of the puzzle for n=1,2,3,4 doesn't matter as much. The fact you START with five as the initial playfield invites the opportunity for collusion that wouldn't exist at simpler n=1 or n=2. Essentially, the decision tree is not well expressed for the more complicated situations.

In reality, this is nothing more than a minimal resource allocation problem dressed up in a sexy way that highlights current income inequality and distribution in modern capitalist economies. It really invites too many messy gray outcomes.

IT projects: it's all about the people

http://www.infoq.com/news/2011/10/risky-it-projects

Well, still no progress on large IT project implementation in the last decade or so. Which compounds on the lack of progress in the previous decades. Of course it's practically impossible to measure projects on the basis of scope, requirements, complexity, etc across the ages. But they can measure the perceived rate of failure and cost.

Software development, delivery, maintenance, and interoperability have not produced major gains, and this study underlines it. I'd argue that the only real broad-scale efficiency success stories have been defined around widely accepted standards, especially HTTP, HTML, and other web standards.

After a couple decades in IT myself, it's pretty apparent to me what gets in the way, and it's not particularly controversial: it's the people, stupid. And that's highlighted by the fact we added several zeroes of capability to the hardware our software runs on, from tens of megahertz to multi-gigahertz processors, from 250 megabytes of storage to 2.5 terabytes of storage (!!). From 4MB RAM to 16 GB of ram. Mother of god those are amazing engineering accomplishments.

First off, lets start with the business people, who control the purse strings and decisions. it astonishes me that still, STILL, to this day, dealing with business people in systems specification at every stage from incubation/prototyping/formal design, indicates fundamental ignorance of IT delivery complexity, concerns, lifecycle, delivery times, etc. They don't even have an approximate understanding. Considering your typical business person has been through the process of getting an MBA, and thus have been "baselined" in the standard practices of business administration, this is pretty telling.

About the most complex thing MBAs learn universally is accounting. IT knowledge is still siloed into the "MIS" degree. And there we have the problem. What business function isn't intimately related to an IT system? Accounting, budgeting, production, supply chain, HR, recruiting, marketing, customer care, I can't think of ANY MBA-concerned business function whose efficient operation isn't tightly coupled to IT systems.

And yet, the basic process, scale, risk, and results from IT projects to integrate, install, configure, adapt, and deliver IT capabilities is treated with absolute ignorance on all these MBAs. Hell, you still, STILL hear about CEOs that proudly state they can't even turn on their computers and have their secretary print out their emails and do dictation. For every dinosaur like that, there are hundreds of people with other stages of malignant IT ignorance, running major companies whose very operation is fundamentally dependent on their IT systems.

I don't get it. You want a competitive edge in MBA land? Have basic-level IT management knowledge. Know about technology at a basic level. Know about the "great divide" between the detail necessary to specify a system expected by non-IT people, and IT people. Know that large-scale IT projects succeed when they represent an evolutionary process, not a dramatic huge system rollout. There's a ton of MBA-level IT knowledge that can be gleaned without knowing how to write a java class, what CORBA is, or any of the other limitless technical complexities and low-level protocols are involved. But your typical business person (as in: 99% of them) are utterly clueless about this.

The second major people problem is access. I''m not just talking about defending your systems from script kiddies and external threats. It's the more insidious beauracracy-generating internal access processes set up by IT organizations. Need to install software? Need to deploy from dev to test to stage to prod? Need access to a machine? Need access to a database? Need a machine to install a server?

Yeah, typical IT organizations you need to navigate a metric crapton of apathetic ticketing systems, signoffs, okays, and manual people decisions to do even basic things. And why? Cover Your Ass. It's all it is. All these managers with oversight over their specific concern (servers, databases, deployments, desktops), rather than enable access and freedom and innovation, would rather lock it down. Really tight. As in nothing goes in or out without them knowing about it.

What causes cross-business systems development failure? Inability to integrate. If you need access to some data (even read-only), you can't just look at the database, slap a quick web service gateway, and then enable the data access. Nope, you'll need to request access to the database, then request access to a new server, request access into the server server, request permission to install applications on the server, request permission on the internal firewall ports, etc etc etc. Sound simple? Typically those are queued behind formal request systems and take days if not weeks to review and complete. Massive time and efficiency losses.

Unfortunately, this is an area I can only complain about, I don't have any good solutions. Perhaps systems need better design from a "future data access" concern. Many/most systems are developed as "service oriented", but rarely are the specifications for the access well documented. Of course most services are probably still written in SOAP, which makes the service convoluted and inaccessible, destroying all the advantages of service orientation.

The third people problem is the IT churn of old-hat new-hat software, protocols, terminology, frameworks, and the like. A lot of these problems are due to the lack of evolved standards and practices that became prevalent over a couple centuries like accounting.

Almost every concept, problem, and concern in IT is represented by a half-dozen terms that largely mean the same thing. Even technologies around well-agreed standards like the internet have far too much terminology variation. This represents a fundamental communication problem even among IT people, who can't communicate concerns, architectures, approaches, and practices as efficiently as occurs in science, law, and accounting.

The fourth problem is the goddamn sales people. This is the most insidious source of problems in IT, since the software sales people talk to the business people, and I can't tell you how many "hey look I built your app in 10 minutes" demoware presentations that completely pull the wool over the businesspeople. Then they get angry when "10 minutes" becomes "10 months".

Businesspeople love being told that all the work done by their underlings that seems so inefficient but usually isn't that inefficient, could be done 100x faster, and they're just morons. The magic bullet. Businesspeople love this, since if the magic bullet works, they get promoted. And all they care about is getting closer to stock options.

There's dozens of these sales jobs underway in large IT organizations at each time, each subsuming and disrupting things right and left. Each basically not solving or enabling anything new.

Because you know what really produces progress? Not magic bullets, but boring evolutionary changes like gradual service enablement, stabilized systems, virtualization.