Using an Interface Engine Efficiently
"Or how I made a bunch of mistakes selling managed integration and why you should avoid them."
If you work for a vendor, you have undoubtedly been asked for more flexibility in how you manage data integration with a healthcare organization. Flexibility could help you due to time, cost or a plain lack of aptitude. This is a reasonable thing to want and interface engines, DICOM routers, ESBs and other similar technologies make this kind of stuff easier. Suddenly you can transform, filter, slice/dice, reroute data with the click/deploy of a finger. Many of these tools have fully-functioned SDEs with modern programming languages, database integration and more. You can definitely “do anything”.
But that definitely doesn’t mean that you should.
When I was doing managed integration work, it was always very easy to say yes to doing custom work in the interface engine we ran for our customers. This was for three notable reasons:
1) In some cases we were billing hourly or some other time-based equivalent, so doing more work made sense.
2) Doing custom work for customers was inherently sticky and meant that they were less likely to leave us as customers.
3) If my team didn’t, someone else would (or would claim to) and we would lose the customer/deal.
Most of the custom work we did for customers was good/logical. Notably building HL7v2 connectors to customer APIs was a good use of an interface engine. It put HL7 data in a place that had tools specifically to manage its transformation and lifecycle and turned junk HL7v2 into clean JSON. However, we also did a myriad of ill-advised things in our interface engine including but not limited to:
1) Generating PDFs from some HTML payload
2) Writing the brains of an appointment reminder system based on variable scheduling inputs.
3) A system that determined whether or not a list of providers received notifications of pathology results or not.
4) Other things that honestly I’ve probably tried to forget due to repressed trauma.
How do you use an interface engine for the powers of good? How do you determine what lives “in code” vs. on your interface engine? Allow me to provide you with some advice.
Just because your interface engine is “Shadow IT” doesn’t mean you should use it that way
When I’m looking at the Mirth Connect Slack or talking to folks about data integration and I see someone trying to do something way too complicated in the interface engine, the first issue that usually comes to mind is that folks do things in their Interface Engine because they simply believe they have no other options to get something done. Either:
1) The process to deploy software at their company is so slow or onerous that getting anything done relatively fast is impossible.
2) The engineering team that controls the stack of the application itself is either non-existent, incompetent or apathetic to customer needs.
3) The folks who run the interface engine believe they know best unilaterally and the other product/software folks don’t know that the interface engine exists so it’s where they can make new things for customers.
These are all reasons. But they are mostly symptoms of a larger disease. You probably have a toxic software-development life cycle. And instead of hacking shit in your interface engine you should fix your people and processes.
I get it. I do. If you’re getting paid for a gig and you’re just trying to make it through the day without Dr. Pain-In-The-Butt telling you and your team what crap your software is and you are personally, anything that allows you to fix the problem fast is a good plan. But most of these fixes likely fall into a few categories:
- Probably complicated and unmaintainable
- Should actually probably be maintained by the customer
- Does not add value to the application
So don’t rush it and discuss the need for new features with your product and eng folks. What if your eng team has zero velocity? Well, I’ve got a news flash for you. If you are a savvy enough of an interface engineer that you could have written some complicated code to solve a problem:
You can write your code in the application instead! That way it follows the rest of your SDLC default processes and code quality mechanisms. Especially now in 2022 there is such a need for tech talent, interface engineers being able to do SWE tasks if they want is such an obvious JRPG class change that you should be actively promoting it within your teams. If you’re not that good at writing code yet in the stack your company uses; learn.
This doesn’t mean that you can’t over-architect things in your interface engine for short-term gains. Interface engines are still a great place to duct tape critical issues to be fully fixed later and prototype new integrations. We’re here to discuss intent though. As such, you avoid writing heavy amounts of business logic in your interface engine vs. your code base when possible.
Then what is good to manage in your interface engine?
Here are some things that are good to manage in an interface engine:
Connectivity semantics involving HL7v2/DICOM/etc.
This is the best thing that an interface engine does vs. writing the entirety of your data integration stack into code. When should you ACK? Should you use MLLP or not? What type of encoding should you support? How long should you retain logs for interface messages when they aren’t the end destination of the software? What is a reasonable time out period? How should you reprocess messages? These things can vary by customer and can also vary by the phase a customer is towards launch. An interface engine gives you every good way to do this vs. trying to figure it out on your own.
Software or messages you never want to receive or retain whatsover
If you only want to accept ADT messages or a subset of ADT messages, the interface engine is a good place to write some logic that says “never send me anything other than ADT messages or I’ll NACK them”. Best way to never apply bad business logic is to ensure that it can never occur.
Standard datatype/string parsing that you’ll always have to do:
Some things about dealing with HL7 are not fun. You can handle them in your app if you’d like but it’s also a good place to handle:
Strip characters/new line structures your downstream software stack doesn’t support
Turn the HL7 timestamps, which are in ISO-8824, into ISO-8601 timestamps with the proper tz offset based on the tz of the integration.
Things you’ll never (or rarely ever) need to touch again after you launch at a health system
Some of this will depend on how much of the data you consume from HL7 messages, but lightweight translation tables that are unlikely to change often are something good to put here. Good examples of this are:
If your software only generate one-type of result but you need something different in the OBX-3 segment at this customer so the data files correctly, just change the OBX-3 segment vs. writing a feature to allow all customers to set the OBX-3 segment.
If the MRN is identified with a certain method in PID-3 in ADT messages you receive, identify the MRN and then do something else with it.
Things that are so weird/bad that they are almost undoubtedly custom work that has no other utility so the way you can get it done the fastest that it doesn’t break is probably fine.
Do you need to dig the patient Care Team out of a Z-segment instead of ROL segments like where they belong? Does the customer want you to write them clinical documentation back in an ORM instead of an ORU or MDM that has OBX segments? Do you have no idea what an acceptable message payload is because you have no specification from the customer and you’re just slamming HL7 messages against the wall to see what sticks? This is what a good interface engine is for.
Once again, you need to be really honest with yourself about this. If a customer also wants you to drop charges and send DFT HL7 messages along with your ORU result messages, maybe that’s not a one-off goofy request! It might be something good to do with your software. But there’s more to good billing than just sending charges, so you should design the feature right.
Maybe I’m wrong.
There’s definitely a season for everything, so maybe there are good situations to hack a bit more than I’m missing. Or perhaps I’m becoming more conservative with SDLC these days. What do you think? Post a comment and let me know.
After a very long hiatus, next blog post will be on understanding Modality Worklists and how to modify them using HL7 ORM messages.