Salesforce and DevOps Part 4 - To DX or not to DX

I have to admit I've been looking forward to this one, after going through the tools and usual lifecycles (and ranting about my views as well).

This post is going to be dedicated to describe my experience, views and decision making process about Salesforce DX, including where and when to use it (or not). Keep in mind that this is all accurate only as of the date of the post creation (August 2018), and as far as I've used it and researched. I'll try to update this one or follow up on different posts in the future. As usual, please comment if there's something you believe I should add or update!

If you want more of a detail about what DX is and some quick start instructions, there's a pretty good blog series on all things DX here: https://developer.salesforce.com/blogs/2018/02/getting-started-salesforce-dx-part-1-5.html. There's also the dev guides, of course.

Let's get to it.

As I mentioned lightly earlier in the series. After a long time of not treating developers (and configurators and testers and architects...) as well as they could and should, Salesforce is starting on the path to make the development experience (i.e. DX) a lot better for all.

After having to live with change sets, manual deployments through IDEs (or just by hand!), ant migration tool antics, it's actually refreshing to see that better ways to implement new features and/or changes in the core platform are coming. 

Even though it has been around for over a year, DX hasn't been widely adopted for many reasons, which I'll try to explain in this post, as clearly as I can.

Things I like about it

There are quite a few things I really like about DX. Don't take my word for them unless you've read the whole post and the things I don't like, though, to get the full picture.

• The concept of scratch orgs: getting to create a fresh environment from scratch is great, especially when you can do it programatically, automate it and know that it will die automatically after a specified amount of time (7 days by default, 30 days max). Also the feeling of killing them on demand is pretty cool as well. They're very useful for developing, running CI builds, playing around with different feature sets and editions and many more things
• With scratch orgs, DX pretty much forces source driven development, and that's a very good thing!
• The use of the CLI: I like CLIs overall, and this one feels pretty cool, as I use it locally and get to see all the things happening on the cloud. It's pretty complete, and plugins like the auto-complete one makes it easier and better to use. There's even a capability to create custom plugins, which I might get into in the near future
• The way that it makes it A LOT easier to push and pull changes. Push and pull commands remove the need to keep obsessive  and granular track of the things that changed on the org and locally. Not needing to use things like the package.xml and ant build.xml makes me happy. This also adds real potential to make it better adopted by functional people, as well as devs. More on this on the things I don't like, though
• The way it simplifies loading data in automatic builds with data plans json files. Remember using the command line dataloader in automated builds? That was fun...
• The concept of developer controlled packages (DCPs) or unlocked packages. Breaking down the source in many packages, while keeping a high level of control on them is pretty cool and I've been wanting to do this for ages (I've done similar things before, but with a lot of hassle). More on this below
• The potential it has for facilitating the achievement of the dream of a fully automated development pipeline (see the end of part 3 of this series)
• The way it simplifies setup of tools for CI/CD, as you reduce the technical dependencies to mostly just having the CLI installed and available
• Keeping a list of authorised production orgs, sandboxes and scratch orgs is quite cool, as you can authorise and open them with a CLI command, instead of looking for your username, password, domain, etc.
• VS Code and the Salesforce extensions plugin is pretty cool when you're on DX, and free to use, I like that
• New debugging features, tail debug logs and actually have breakpoints and so on, probably not necessarily related to DX, but it is related to the CLI
• All in all, getting new things to play with is always cool, and if it evolves fast and steady, it would make Salesforce delivery a lot more streamlined and just more fun

Things I don't like about it (at least yet)

Ok, if you see the list above on it's own, you'll probably think I'm a DX evangelist and super optimistic (or perhaps you chose other words) about it. But even though I am reasonably optimistic about the future of DX, I'm not sure it's still a full product, unfortunately. 

List of things I don't like (at least yet):

• Metadata API completeness: let's get this out of the way first. Every Salesforce deployment method is based on the Metadata API. Even more, DX is based on the packaging capabilities, on top of the Metadata API, which limits it further. As if that wasn't enough, unlocked packaging is limited even further. The following are some annoying things I've found with it:
• Profile issues: the big one! As with the metadata API, managing profile changes are a pain, it's really really annoying. Example of the things you might encounter: you change things in your org, you pull, you see the file changed, but it doesn't include what you did, for some reason. Use permission sets, please!
• OWD: similar to profiles, I've found that sharing settings are buggy to maintain
• Communities: problems getting the metadata from a community using standard builders. Custom sites are fine, to a point
• Lead conversion mappings: sometimes you can push changes, but pull doesn't seem to work
• There are likely many more, for sure, please comment on the things you've found!
• Scratch org definitions: getting a definition to create scratch orgs that is the same as you org takes some (or a lot of) work, and more often than not, you'll find that you missed things the first few times you use them. It complicates deployments, and therefore delays releases. For example, sometimes you'd think you've finished something as it works in the scratch org and on CI builds, just to find that it doesn't really work in your sandboxes and / or production org. I know there's a pilot for something called "org shape" which should address it in the near future, but I'll have to see it to believe it, I guess. There's also an npm package called hydrate (also an sfdx extension) that you can use to do this, feel free to check it out, it might work for you!
• The non-existence of "sandbox like" scratch orgs that would come with metadata (and maybe data) from production or from a sandbox, and that you could interact with using push and pull commands, create and kill programatically and so on
• Complementary to the previous point, pulling down metadata from an existing org can take a long time. To prepare your repo/project to start working with DX on an existing org, you have to use any of the old fashioned ways to do so and run the CLI's convert command to transform it (at least there's that!). Think package.xml creation, retrieve, transform, trial, error, repeat!
• Dependencies with managed packages are sometimes too hard to overcome. In line with the previous points, when creating scratch orgs, sometimes installing some packages is needed. Although there are ways to programatically do this (CLI install commands, adding the installedPackage metadata types to the directory), some things like vendors having passwords for their packages that they're not willing to share and complicated configuration that needs to happen after package installations, among others, makes it uphill to fulfil the dependencies in a useful way
• The way that sometimes the change history for pulls and push gets corrupted and there's a need to force flag the commands, which actually makes it worse most of the times. The alternative of using the force flags on push and pull commands is to go to the metadata files in the hidden .sfdx folder, but at that point it's a real trial and error kind of thing, which really fastly evolves into killing the scratch org and starting over
• The time it takes to push! seriously, as the number of things in you metadata grows, it gets a bit ridiculous sometimes...
• Unlocked packages (DCPs in all) is a beta feature. It has been for a while, which makes it a bit buggy sometimes. The following bug stopped me from using them altogether in the past, while working on a managed package implementation: installing an unlocked DCP would successfully finalise and the functionality got there as usual, but the Apex classes in the target org were hidden (literally, when opening the class, they only had the text "(hidden)", which I didn't notice at first, given that testing passed correctly). You can imagine how trying to install a managed package created out of that worked out. If you can't imagine, I'll tell you, it didn't! When trying to install the created managed package it kept failing due to unexpected "(" characters, at which point I went to the classes and found the "(hidden)" issue. That's some time I'll never get back. Happy to say that I just found out that this bug is gone now, unless you're creating a managed package and working with namespaces, in which case it's still there, code is still hidden
• The time it takes to create unlocked package versions: even with just 2 or 3 files in the package (I've tested this!), it takes a few minutes (5-10, usually) to create a package version. Seems small, but feels forever, think of what that would look like in an automated build
• Dev Hubs are just available in production orgs. Not a deal breaker, but still annoying to maintain, as everyone needs a user in production, if they are to create scratch orgs and packages and so on. There are permissions for that, to avoid giving access to unnecessary things, but still. Would be really nice to have a different option

When to use it and when not to

In my view, if you and your implementation can fully relate to the following points, I think you should give DX a go:

• You can base your security on permission sets, not using profiles except for the fact that they're required at the moment. I find that I actually like this principle either way, just cloning the read only profile a really small number of times (one for business users, one for service users, for example), an then build everything else on top with permission sets works quite well. Of course, certain things need to live in the profile, like layout and record type assignments, so plan accordingly and decide on your conventions wisely. Rumour has it that Salesforce will eventually deprecate profiles, or at least upgrade them or give us the option to work on something that uses permission sets tech (they called this permission set group last Dreamforce), that would make me happy, let's see how long it takes...
• You have no dependencies on managed packages. At the very least, you can install them without passwords and they don't require manual configurations after installation for your current implementation or package
• You are in a completely brand new org, with a greenfield implementation. To avoid most of the issues with scratch org definitions and getting the metadata down in the first place. In this case, it actually works quite nicely, from my experience
• Your implementation is for an ISV package and / or it's mostly code (Apex, Visualforce, Lightning components...). Ive used it for this, on a complex package (lots of code, some lightning components, integration with APIs and so on), with no real issues, pretty cool!

When not to use it (sadly):

• Every other scenario! Unless you're happy to take on the challenge, the cost and the time it takes to manage certain things with DX, certain others with different methods, lots of manual steps, difficult maintenance and more, of course

About Unlocked packages and all DCPs (they're still BETA)

I wanted to add a specific section on these, as I find them really useful and a great idea. Of course, always with the caveat of the limitations and things I like and don't like, mentioned above.

For some context, the following things are what I believe will make them super useful:

• Having one "common" or "base" package that contains metadata that all work streams will need, that can be managed/implemented by a team on it's own
• Creating packages that only depend on this "common" one but that address specific issues, features. Each of these could then be managed/implemented by a specific team in it's own work stream, if needed
• Removing the need for destructive changes. They will be intelligent enough to deprecate items you remove from your package, and good riddance to them!
• Working on packages in isolation, even in different orgs. DCPs can be installed in any org, regardless of the relationship with the Dev Hub

I find it super annoying that, while unlocked packages are not generally available (and complete enough), we have to use DX just for the "dev train" and switch to metadata API deployments for the "release train"! I know there are CLI commands to convert and deploy/retrieve, but still. Having said that, when this is possible, it's still better than before DX, in my opinion.

Considerations, things to be aware of and things I don't have answers for just yet:

• Given that existing orgs can be monolithic in the amount of metadata they have, and how it's structured, it can be a tough job to break things into different packages (DX's version of microservices, perhaps? There might be something there for the future, in conjunction with platform events, for example). In any case, I don't recommend a big bang approach for doing that. In a nutshell (probably oversimplifying it), you should do something like this:
• Every new implementation should be done with packages, while considering packaging the dependencies as a starting point for building a "common" package
• Little by little, as available, try to do one package at a time for the whole "monolith". This could be a project on it's own right, almost, so be careful
• How to manage different packages in source control, do we use one or many repos? Given the uniqueness of metadata components (a component can be in one and only one package), if we decide to go for one repo, there needs to be very good communication and conflict resolution capabilities in your implementation. If using one repo per package, then dependencies become harder to maintain and conflicts can increment. In both cases, adapting the automated builds to sort out dependencies is key
• It's a very good and useful thing that packages can be created, versioned, installed, updated and deleted programmatically, but I still haven't quite figured out the way this will be used in conjunction with the development lifecycle (as explained in part 3 of my series). Things I have left to include and work on (will update as I go along):
• Automation of package creation (maybe not?)
• Automation of package VERSION creation (definitely, but soooooo slow)
• Adapting branches, steps, testing and artifacts to work with packages (don't think it'll be too different, though)

Conclusions

Some thoughts to wrap the post up:

• We need to be using and adopting DX a lot more, and that's a fact. Even though it's not always possible, as you've seen, I hope that Salesforce can helps us, fast!
• All in all, things seem to be evolving towards automation and source driven development in Salesforce, which is nice to see (even though it's years behind)
• Salesforce is pushing for improvements to the developer experience. To the point that, ever so slowly, you'll find that even events like Dreamforce and the like feature it more and more each time
• Salesforce really seem to be concerned about metadata API coverage (this time it really looks like it, I guess). The Metadata completeness report that they released this summer is a good sign, as it helps to easily identify, specific for each org, which things are covered and which are not. Check it out here: https://[yourOrgUrl]/mdcoverage/report.jsp
• There's a good community behind SFDX, check it's chatter group

Give DX a shot and please let me know how you got along with it, I'd love to hear your stories and experiences with it

Next in the series: wrap up, future predictions for Salesforce and DevOps and my personal wish list