Sir, I’m gonna need to see UUID.

Said the stuttering policeman to the wayward Mac.

Today I encountered a very strange issue with Jamf, and the root cause was not what I expected.

TL;DR: a corrupted computer record caused an advanced search to fail because the value for one internal field—a field whose values should never, ever have bad data—somehow got bad data.

BACKGROUND

Another engineering team within our company reported that a daily Jamf feed to a very important third party inventory service had stopped running since about a week ago. Running the report via the API resulted in HTTP 500. Downloading it via the Jamf console resulted in a blank page almost instantly.

The Jamf logs were returning an error without debug enabled, but this did not tell me what was wrong. Rather than open a support case and wait, I decided that diving in would be faster.

We have an on-premises Jamf cluster, which affords us the ability to perform some very detailed analysis of the logs and the database tables if necessary. I logged into one of our “API only” Jamf nodes. This is inside our company network and connected to the other Jamf nodes in the cluster, but not behind the client-facing load balancer. Therefore, managed endpoints cannot reach it. We reserve this endpoint strictly for running reports and deleting computers, a task which has historically taken a very long time in Jamf, in order to keep the client-facing nodes available.

THE ERROR OF OUR WAYS

Here’s the relevant portion of the error message:

2022-05-23 18:22:12,645 [DEBUG] [Thread-4] [com.jamfsoftware.jss.frontend.HTMLResponse] - Processing parent action list to populate navigation: ComputerHTMLResponse
2022-05-23 18:22:12,645 [DEBUG] [Thread-4] [com.jamfsoftware.jss.frontend.HTMLResponse] - Responding to custom action: DownloadReport
2022-05-23 18:22:12,645 [DEBUG] [Thread-4] [com.jamfsoftware.jss.objects.CRUDHelper] - Verifying READ privileges for Advanced Computer Search
2022-05-23 18:22:12,661 [DEBUG] [Thread-4] [com.jamfsoftware.jss.objects.advancedsearch.AdvancedComputerSearchHTMLResponse] - Performing advanced search…
2022-05-23 18:22:14,578 [ERROR] [Thread-4] [com.jamfsoftware.jss.objects.advancedsearch.AdvancedComputerSearchHTMLResponse] - There was an error calculating the advanced search:
java.lang.IllegalArgumentException: Invalid UUID string:
at java.util.UUID.fromString(UUID.java:215) ~[?:?]
at com.jamfsoftware.jss.objects.computer.ComputerHelper.createComputerFromDenormalizedRecord(ComputerHelper.java:3366) ~[classes/:?]
at com.jamfsoftware.jss.objects.computer.ComputerHelper$3.process(ComputerHelper.java:2090) ~[classes/:?]
at com.jamfsoftware.jss.objects.computer.ComputerHelper$3.process(ComputerHelper.java:2085) ~[classes/:?]

[...]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]

For all its faults, Java is pretty easy to debug because of the stack trace: the long chain of Java classes loaded in memory, leading back from the initial faulting module to the one that started the current chain of events. Sometimes it can be a bit overwhelming, but its thoroughness is its saving grace.

Breaking it down:

  1. The action of “Performing Advanced Search…” threw an exception (caused an error).
  2. The exception was thrown by “java.util.UUID.fromString.”
  3. This Java utility was called by the ‘ComputerHelper.java‘ class.

Ostensibly, the exception occurred because the data being fed to the utility was out of bounds or out of range for what it was designed to do. For a string converter, the problem can be as simple as invalid characters, a totally empty string, etc. For instance, a “stringToNumber” utility will convert the string “23” into a numeric value of 23, but it will throw an exception for the string “twenty three.”

…and that is as far as we will take the theory. The purpose of this blog entry is NOT to analyze java.util.UUID.fromString, but to figure out what’s happening in Jamf to cause this.

A PEEK UNDER THE HOOD

All Jamf computer records start in the Jamf database. Thus, the advanced search / report is choking on one or more records in the database. Now, Jamf has a lot of sanity checks to ensure good data always goes in, so you don’t get ‘garbage in, garbage out’ problems later on. But the most glaring problem with some of Jamf’s error logs is that there is no telling which computer record might be causing the issue.

Thankfully, since we are an on-premises instance, we have direct access to the database AND the Jamf classes that power the whole thing!

In a typical Jamf installation, you will find this folder structure:

D:\JSS\Tomcat\webapps\ROOT\

This is where the guts of Jamf actually exist. “Tomcat” is like the ignition, steering wheel, and gas pedal in your car. You turn it on, point the wheel, and press the pedal, and it goes somewhere. The parts that actually make the car go: engine, spark plugs, alternator, radiator, various pumps, plumbing, belts, fuel lines, sensors, battery and electrical … all of that are the front-end things like Java Server Pages, and back-end things like Java classes and libraries.

We’re going to be looking specifically at:

D:\JSS\Tomcat\webapps\ROOT\WEB-INF\classes\com\jamfsoftware\jss\objects\computer

Now, looking back at the log:

com.jamfsoftware.jss.objects.computer.ComputerHelper.createComputerFromDenormalizedRecord(ComputerHelper.java:3366) ~[classes/:?]

Each period is one level in the Java class hierarchy, from root to branch. Sort of like folders in a file system.

So if we look at the folder structure…

Whoomp. There it is.

TAKING IT APART

Now, how do we actually read the class?

The Jamf log gave us a line number, i.e.: ComputerHelper.java:3366. Java classes are traditionally compiled into binary (machine language) for fastest execution. So you can’t just open them in a text editor. You need a Java decompiler.

Fortunately, Java decompilers are easy to find and often free, and they all do pretty much the same work, from the simplest online Java decompiler to a full blown IDE like NetBeans or Eclipse. Whatever you prefer. But all will yield something like this:

Now if you’ve ever written any code before, this is starting to look familiar — importing libraries, creating your functions, etc…

So the next step is to look at the method where this call failed: createComputerFromDenormalizedRecord.

There’s the method name in 2552. Does line 2557 look familiar? “UUID.fromString.”

As far as I can tell, this particular method takes values and maps them to fields in the database. And the purpose of “throws SQLException” is to prevent garbage data from getting in. At least, in theory…

So now I know this function is converting the UUID into the field management_id.

But which database table?

ON CLOSER INSPECTION

The other piece that I recognize is “denormalized.”

In data storage, the purpose of denormalization is to increase efficiency of routine searches by having some redundant data available in a related table. Rather than create a complex series of JOIN statements, denormalizing can make the most commonly accessed data available in the same table, thus decreasing recall time.

Jamf has two tables containing computer data: the main computers table, and a partially-redundant copy called computers_denormalized.

To examine these tables, you will need the credentials to your Jamf database, and appropriate access set to the jamf database user in MySQL.

From the MySQL command line utility:

  1. /usr/bin/mysql -u username -p [Enter]
  2. Enter the password.
  3. use jamf_database_name
  4. describe computers_denormalized;

The denormalized table contains a little over 1KB of data for each computer record. But even for an ‘optimized’ database, this table has over 50 fields. It’s too much to look through efficiently, and I’m only really interested in the JSS ID computer_id, computer_name, and of course, the management_id :

select computer_id, computer_name, management_id from computers_denormalized;

OK, this is starting to look helpful.

The management_id contains only the characters 0-9 and A-F.

At this point, we can speculate that the management_id throwing the original error does not meet this requirement. So let’s find it…

Confession: MySQL is not my strong suit, so I had to look up the syntax for regular expressions, and I validated this with www.regex101.com (a very useful site for learning and testing Regular Expressions).

select computer_id, computer_name, management_id
from computers_denormalized
where management_id not regexp '[0-9]|[A-F]|-';

Et voilà !

That’s definitely not supposed to be empty!

FORENSIC ANALYSIS

So know we know what caused the problem. But why did it happen?

When I checked this computer record in the Jamf console, it pulled up a seemingly normal record with a hostname and hardware data.

Except … there were little things not quite in order:

Why is the serial number “Unavailable?”

Yeah, this is definitely not right.

Why is the policy log blank?

The computer usage is still populated!

CONCLUSION

So… we have a partial computer record with past usage data.

My best guess is that a re-enrollment did not complete successfully.

The only solution here is to delete the offending computer record. Thankfully, you do not need the database to accomplish this. Deleting the record using the “Delete” button will purge the entire record out of all tables.

Sir, I’m gonna need to see UUID.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s