Wired Thoughts Mac OS

Computer compatibility. Mac OS X V10.6 and above including new Mac OS Sierra. any desktop or notebook computer with an available USB port worry-free warranty At Macally, we believe in our products. That's why we back them all with a one-year warranty and provide friendly, easy-to-reach support for any of your questions. Restart from OS X Recovery by holding down Command-R while your Mac starts up. Select Disk Utility from the list of OS X Utilities. Select your drive from the list of internal drives in the sidebar. The default name is “Macintosh HD.” Your drive might have a different name or location, if you renamed or moved it.

Introduction

Metadata is simply 'data about data.' Within the context of the computer industry, the most common domain of metadata is the file system. Files contain data, which has some amount of associated metadata. The most fundamental property of metadata is that it is distinct from the data itself. Again, that is the very definition of metadata: data about data.

It's a simple concept, but it's also a surprisingly fertile topic for debate. The introduction of Mac OS X has brought this topic to the forefront of the larger battle for the future of the Mac platform. To understand why, we'll examine the fundamentals of file metadata (or simply 'metadata' henceforth), and explore the past, present, and future of metadata on the Mac, and in the computer industry as a whole.

There is one important assumption to keep in mind. This article will generally confine itself to the contemporary concept of files and file systems. Object-relational storage or other such divergences from the familiar files and directories common today will be touched upon in later sections, but will not be considered in the main body of the article in order to keep the focus manageable.

There are also some assumptions to avoid. The first part of this article will talk about the fundamental concepts of metadata, irrespective of any implementation issues (with the exception of the single, focus-narrowing assumption listed above). While reading about the fundamentals of metadata, technically inclined readers (that's everyone who reads Ars, right? ;-) may find it very difficult to avoid slipping into thoughts of actual implementations they are familiar with. Don't do this. Try to concentrate on the concepts being presented without making judgments or coming to conclusions based on particular implementations or standards you may be familiar with. We'll get to all that later in the article, after the fundamentals have been established.

Fundamentals

To illustrate the fundamentals of metadata, let's start with a hypothetical file. Without any metadata, this file is just data: a bunch of bits. What kind of 'data about this data' can we come up with? Let's make a list.

  • The file's name. This is an essential piece of metadata, given the assumption listed earlier regarding the concept of a file. A file's name is an essential part of the access mechanism in a traditional file system.
  • The file's location. We need more than just the name to uniquely select a file in a traditional file system. A file's location is some combination of the host, disk, and directory structure where the file is located. A file may be uniquely selected by combining the file's name and the file's location into a single identifier (often called the 'path' to the file).

    Most people are not accustomed to thinking about a file's location as a piece of metadata associated with the file itself. This is your first test: are you thinking about 'data about data' at the conceptual level, or have you already slipped into an implementation mindset and snorted derisively at the idea of file locations as metadata?

Think about it. A file's location is certainly 'data about the data.' In fact, it's essential data about the data. (We'll make some distinctions about different types of metadata later.)

  • The nature of the file's data. What does the file's data represent? What is the file? Is it an image? A movie? Text? This concept is also called the file's 'content type' or just the file's 'type.' There are many levels of detail possible, from basic classification (image, audio, video, text, etc.) to specific file 'formats' (JPEG, AIFF, MPEG2, Microsoft Word, etc.), or even very specific incarnations of particular formats (GIF89a, Photoshop document with layers, Microsoft Word 6.0/95, etc.)
  • The file's size. The data in a file has a size, even if it's zero. Again, this is not what most people think about when they think of metadata, but it most certainly fits the bill. Like file name and location, it's another essential piece of information about the file. (Imagine the limitations of a file system where a file's size was unknown!)
  • File dates. There are many useful dates one can imagine associating with a file: creation date, date of last data modification, date of last metadata modification, date of last data access, and so on.
  • File permissions. Like file dates, there are many possible variants of this metadata. Who can read this file? Who can write to it? Who can run it, if it's executable? Etc.
Advertisement

I'm going to stop the list now, but not because it's in any way complete. I'm sure you can think of many more pieces of metadata that are applicable to a file, especially if you focus on metadata that may only be applicable to particular types of files. For our purposes, I just wanted to cover a set of basic metadata that spans a conceptual range, which we'll explore next.

Types of Metadata

Take another look at the short list of file metadata we came up with:

  • Name
  • Location
  • Type
  • Size
  • Dates
  • Permissions

If you had to divide this list into groups, how would you do it? Are there certain pieces of metadata that share common traits? Think about it before continuing.

<pause for thought>

Let's start with permissions. This is metadata in the traditional sense familiar to most people. It describes the access rights for the data (in some implementation-specific way that you shouldn't be concerned with right now, remember? :-)

The nature of file permissions as metadata is best illustrated by changing them. Despite the fact that file permissions are indeed data about the file's data (who/what can do XYZ to this data?), they may be changed without regard for the data itself.

What other pieces of metadata from our list share this quality? Name and location seem to fit the bill. Changing a file's location, for example, does not imply a change to the file's data.

Some file dates also exhibit this quality. Creation date and possibly the last access data (assuming read-only access) also may change with complete independence from the data that they are associated with. (Modification date, on the other hand, should not change unless the data itself has been (duh) modified.)

We'll call these pieces of metadata (name, location, permissions, and some kinds of dates) 'independent metadata' to signify their ability to be changed independent of any changes to the data itself.

We've already grouped more than half the list ('three and a half' out of the six) into this category. Independent metadata is the most common type of metadata. Almost all metadata can be changed without requiring changes to the data itself. But independednt metadata is not necessarily the most important type of metadata. Let's look at another type, using file size as our first example.

Size is an essential piece of metadata, but like location, it is rarely thought of as metadata. Why is that? Perhaps it's because it's so essential that its presence is taken for granted. After all, if the extent of the file (i.e. its size) cannot be determined, the file is nearly useless! For example, how would the operating system know when to stop reading data from a file without any information about its size?

A file's size is an example of what I'm going to call 'immutable metadata' in that in that it is metadata that cannot change for any given set of data. In other words, immutable metadata may not change unless the data itself also changes. Immutable metadata can also be thought of as 'data-dependent' metadata, but I have chosen to call it 'immutable' to emphasize that it cannot be changed directly. The immutable metadata associated with a given set of bits (i.e. data) is indeed immutable. The only way to get different immutable metadata is to get different data.

Advertisement

Think about the consequences of violating the constraints of immutable metadata. Imagine a scenario in which the 'size' metadata for a file was changed without any change to the actual data! Immutable metadata is worse than useless if it is not completely tied to the state of the data itself.

The concept of a file's size as 'immutable metadata' may be difficult to get your head around, particularly since we're so used to thinking of a file's size as something that is 'derived from' the data itself, rather than simply 'associated with' the data like the independent metadata described earlier. But that's the very definition of immutable metadata: it is metadata that is inexorably tied to the data itself.

Modification date was touched on earlier, and is another example of immutable metadata. The modification date is changed in response to an accompanying change to the data itself. Changing a file's modification date without changing the data itself is not as disastrous as doing the same thing to a file's size metadata, but it is certainly not 'correct' behavior.

Finally, there's the file's content type. This too is immutable metadata. A file's content type, by definition, cannot change unless the data itself also changes.

Thoughtful readers may have already realized that there is one case where immutable metadata may change without requiring a change to the data itself. Immutable metadata may change to increase (or decrease, of course) its accuracy. For example, a file may have associated file type metadata that identifies it as a GIF image. At some point in the future, it may become known that the file is actually an interlaced GIF89a. The file type metadata may be changed to reflect this increased accuracy without requiring a change to the data itself. Similarly, a file's modification date may be increased to millisecond accuracy, and so on.

This leads to some more general rules for metadata. Like all forms of information, more is generally better. More metadata with greater detail makes more interesting decisions possible. Without permissions metadata, for example, it is much more difficult to make intelligent decisions regarding access to a file's data. And if a file's creation date only includes the year, it becomes impossible to distinguish the relative ages of files created within a given year. Information is power.

One final note on metadata which may seem obvious, but which will become a factor later on. Like all forms of information, metadata is easy to remove or ignore, but it is often difficult (or impossible) to add once it is lost. If you no longer know when a file was last modified, you cannot recover that piece of information (despite the fact that the modification date is immutable metadata completely tied to the data itself). The data itself remains, but the information about when that data was last modified is lost. To use an analogy, think of lossy compression. You can convert a CD track into an MP3 file, but you cannot convert an MP3 file back into the uncompressed audio from the CD. You cannot 'add' unknown information. Information is precious.

While reading thru some Apple forums, I found a question people asked a lot. While most responses were generally correct, I felt that all answers lacked some important details and as a result confused people even more. The purpose of this article to clarify those details as much as possible.

First, let’s answer the first question from the title of the post.

What is wired memory?

Wired memory on Mac is reserved for the macOS kernel and its data structures, and it cannot be paged out. The normal size of the wired memory is around 2GB. User applications add on average 1MB per instance, and too many running programs can contribute to high wired memory usage.

Now, let’s dive into more technical details.

When the Mac suddenly becomes too slow, the laptop owners start browsing forums and learn to check the memory tab in Activity Monitor.

The screenshot above which I made after rebooting the Mac indicates that my Mac has 8GB of Physical Memory and 2.33GB of it is in the Used Memory bucket. The used memory then consists of App Memory (1.2GB) and Wired Memory (1.13GB).

Wired Thoughts Mac OS

Why do people ask about Wired memory?

Wired memory is a memory that can’t be compressed or paged out to your startup drives, so it must stay in RAM. Moreover, other processes can’t borrow it.

The key phrase here is “can’t be paged out.”

Before we proceed further let me explain what paging is.

Wired Thoughts Mac Os X

The architecture of modern computers was designed in the 20th century. Any computer has three main parts: processor, RAM (random access memory) and storage (disk, tape, punched cards, etc.).

The processor works directly with RAM because the latter is thousands of times faster than HHD or SSD. Besides the speed, there are two more differences between RAM and HHD:

  • RAM needs constant power. When a computer shuts down or restarts, the data in RAM gets wiped. Hard drives preserve data even when the computer is off.
  • RAM is much more expensive. For instance, for an additional $200, you can buy either 8GB more RAM or 128GB more SSD when purchasing a new MacBook Pro.

The last point is significant. If RAM was cheap, then Apple would stuff its laptops with tons of memory, and nobody would care to read this article. But since its more expensive than disk storage computer engineers invented memory swapping or paging.

Let me use a whiteboard analogy.
Imagine, you work in a startup company which rents couple desks in a coworking space. Amenities in a coworking environment include unlimited coffee and beer, Wi-Fi, bike storage, and a whiteboard.

When Team A needs to brainstorm the Next Big Thing, they go to the whiteboard and start writing down some ideas. Unfortunately, there is only one whiteboard on the floor. Team B wants to brainstorm their TNBT too, and they also need some whiteboard space.

Team A graciously agrees to share some of the whiteboard space. Then another team comes, and another. With each team, the amount of free space shrinks.

When the available area gets too tight, teams decide to write down some of their ideas on the paper and clear the piece of the whiteboard for new ideas. When they need to access their old ideas, they search thru past papers and copy them to the whiteboard.

Memory management in modern operating systems works similarly. In this case, a whiteboard is a physical memory which is shared by apps (teams). When apps request additional memory, and there is no more free space, the OS checks all memory pages.

The OS tries to find ones that were not used recently and copies them to the disk. Apps do not write down unused memory like in whiteboard example, the OS does. When the app needs back that data, the OS reads it back to RAM from the disk.

In theory, more disk storage the computer has more RAM can be extended by paging. The reverse is also true, in case of little or no free disk space swapping is almost impossible.

This wonderful mechanism, however, has a significant flow: disk is thousands of times slower than RAM. When there is no enough memory, the system keeps paging in and out data continuously, which causes so-called disk thrashing.

In the latter case, the computer, instead of executing programs keeps moving data making the life of a computer user miserable.

So what it has to do with wired memory? Continuing the analogy, imaging the coworking administration decided to block the quarter of the whiteboard and write a schedule of events on it. Plus they disallowed any team to wipe off the schedule.

More space the administration takes less whiteboard space teams can use. More wired memory the macOS uses fewer memory apps can share.

Mac Os Download

By the way, if you were wondering what represents Cached Files in the first screenshot, it’s a memory that was used by an app which was recently terminated. It’s as if Team A decided to go to a nearby restaurant to grab some lunch.

If by the time they come back, their ideas are still on the whiteboard, they can start from where they left. If not, then they have to copy everything from their papers back to the whiteboard.

Cached Files is a mechanism which macOS uses to speed up the recently closed applications in case if a user wants to start them again. If meantime another app needs more memory, the OS will claim it from Cashed Files.

So, if you see too much Cached Files don’t worry, it’s same as free memory.

If you need proof, try following in Terminal:

  • In the Terminal app type sudo purge and hit Enter
  • Type in the admin password

When I ran this command, the only thing that has changed in Activity Monitor was Cached Files: its size dropped from 1.89GB to 594MB.

Is it ok to use sudo purge? There is no harm because all it does is clear unused memory. But it also doesn’t do anything valuable. macOS can release unused memory when apps ask for more.

What’s included in the Wired memory?

According to Apple documentation, wired memory is a part of memory reserved for OS kernel and its data structures. Following kernel objects belong to the wired memory: Virtual Memory objects, virtual memory and I/O buffer caches, various drivers.

User applications do not directly work with the hardware; they send requests to kernel drivers and those, in turn, perform hardware operations. For instance, the internet browser does not send pixels to the MacBook screen. Instead, it sends requests to the video driver, which then changes colors of pixels on the screen.

Wired

However, user apps can also influence how much of wired memory is used, although indirectly.

Following is a table with some (not all) types of objects user apps indirectly create in the wired memory. Note that the data in the table is outdated, but it at least gives some idea why the wired memory changes every time we load or unload an app.

Let me explain some of the objects in the table on the example of the Google Chrome browser. When I first started Chrome, it created several processes: one Google Chrome and 8 Google Chrome Helpers. The wired memory immediately jumped to 1.35GB.

Each line in the Activity Monitor is a process: an application instance. Each process is a collection of threads and threads perform actual work on the computer. If data in Table 1 was current, we could count that Google Chrome process added 16KB to store process data, plus 46 * 21KB = 966KB for threads.

Chrome is an interesting app. Every time you open a new tab, it starts additional helpers. Each helper is a separate process, and each process spawns numerous threads.

When I opened one tab, the number of Chrome helpers jumped from 8 to 16. After opening a second tab the number of helpers increased to 24, and after third to 26. After opening three tabs, the wired memory size increased to 1.61GB.

Wired Thoughts Mac Os Catalina

As you can see each process indirectly increases wired memory because macOS has to record those threads somewhere.

And this is not all. While Apple claims that wired memory is reserved for kernel and some objects which describe user processes, some applications can tap into the wired memory directly.

For instance, if you ever used virtualization software such as Parallels Desktop or VMWare, you should know that they use the wired memory directly to improve the performance.

If you think about this, it makes perfect sense. Virtual frameworks are designed to act like real machines so that OS running inside can handle the memory the way they always do. For instance, if you run macOS virtual machine on Mac, the macOS will create its version of wired memory inside VM.

After digging a bit in the Kernel Programming Guide, I found that applications can allocate memory in the kernel by calling kmem_alloc_wired function instead of regular alloc.

Now, what all of this means in practice?

How to free up memory on Mac

There are two main reasons for the high usage of wired memory:

1) Some applications modify the kernel and use wired memory deliberately. Examples are drivers and applications working on the kernel level such as antiviruses, or virtual machines. If for instance, there are some widgets in the Mission control, turn off those you don’t need.

In the case of virtual machines, the remedy is to limit the memory configuration. For example, if the Windows VM is configured to use 8GB of RAM lower it to 4GB.

2) When too many applications are active, they still use RAM. Unlike Windows, where clicking on close button unloads the app, in macOS, the app continues staying in memory. You can tell them by a little dot under the icon in the Dock.

Use Command and Q key combination to quit the app altogether. If some apps still stay in the memory rebooting the computer sometimes can fix the problem.

And finally, high wired memory usage is not always a sign of a problem. In the following example used memory was almost 31GB on a 32GB Mac, and wired memory accounted for the bulk of used (26.61GB).

Mac Os Catalina

However, look at the left side of the screenshot: Memory Pressure. Memory Pressure section in Activity Monitor can have one of 3 colors like a traffic light: green, yellow, and red. If memory pressure color is red, then more RAM is needed. In case if it’s green, then there is enough memory for the current load.

Wired Thoughts Mac Os X

Wired memory does not get released immediately when an application quits. It gets cleared by a garbage collection process which may take some time, and during this time the wired memory size may be too high.

Resources:

Image credit: Flikr