When working with the SharePoint REST API, developers often encounter a common hurdle: the sheer volume of metadata. If you have ever queried the /_api/Web/Lists/GetByTitle('MyList')/Fields endpoint, you know that SharePoint returns a massive array of fields. While this is comprehensive, it includes a multitude of system-level columns, hidden fields, and internal identifiers that your end-users never see.

In this guide, you will learn how to refine your SharePoint REST API queries to target only the columns that matter. We will explore multiple techniques to filter out the noise, ensuring your application only interacts with the user-defined columns visible on forms and views.

Understanding the Fields Endpoint Noise

By default, the Fields endpoint returns every property associated with a list's schema. This includes internal plumbing like LinkTitle, ContentTypeId, ScopeId, and many others. If you are building a custom form, a data export tool, or a dynamic table, these system fields can clutter your UI and degrade performance.

To clean up your data, you need to leverage OData query parameters. The primary tool at your disposal is the $filter parameter, which allows you to define specific criteria that a field must meet to be included in the response.

Method 1: Filtering by Visibility and Read-Only Status

The most straightforward way to get columns that a user actually interacts with is to filter based on whether the field is hidden and whether it is read-only. In most scenarios, user-defined columns are neither hidden nor read-only (unless they are calculated fields).

Here is the standard REST query for this approach:

/_api/web/lists/getbytitle('YourListTitle')/fields?$filter=Hidden eq false and ReadOnlyField eq false

Why this works:

  • Hidden eq false: This excludes internal SharePoint fields used for tracking (like Author or Editor in some contexts) and GUIDs that are never intended for the UI.
  • ReadOnlyField eq false: This excludes fields that the user cannot manually edit, such as the ID field or system-generated timestamps.

Note: While effective, this method might still include a few system fields like 'Attachments' or 'Content Type'. We will address how to handle those in the following sections.

Method 2: The Advanced SchemaXml Filter

If you need a more robust way to distinguish between "System" fields and "User-defined" fields, you can look at the SourceID attribute. SharePoint identifies its own internal fields using a specific namespace, while user-created fields are assigned a unique GUID.

All system fields in SharePoint 2013, 2016, 2019, and Online typically have a SourceID of http://schemas.microsoft.com/sharepoint/v3. By checking for a GUID instead, you can isolate the columns created specifically for that list.

/_api/web/lists/getbytitle('YourListTitle')/fields?$filter=substringof('SourceID="{', SchemaXml)

Why this works:

This query uses the substringof function to inspect the SchemaXml property of each field. User-defined fields contain a GUID wrapped in curly braces (e.g., SourceID="{GUID}"), whereas system fields contain the schema URL. This is arguably the most precise way to identify custom columns.

Method 3: Combining Filters for a Clean Dataset

For the best results, many developers combine the visibility filters with specific exclusions. This ensures you get columns that are editable, visible, and relevant to the business logic of your application.

Here is a comprehensive query that selects only the most important properties and filters out common system outliers:

/_api/web/Lists(guid'00000000-0000-0000-0000-000000000000')/Fields?$select=Title,InternalName,TypeAsString&$filter=Hidden eq false and ReadOnlyField eq false and InternalName ne 'Attachments' and InternalName ne 'ContentType'

Key Components of this Query:

  • $select: Always limit your selection to the properties you need (e.g., Title, InternalName). This reduces the payload size and improves performance.
  • InternalName ne 'Attachments': Explicitly removes the Attachments toggle if it isn't needed for your UI.
  • TypeAsString: This property is incredibly helpful for determining how to render the field in your application (e.g., Text, Note, Choice).

Best Practices for SharePoint REST Queries

When working with list metadata, keep these best practices in mind to ensure your code remains performant and maintainable:

  1. Prefer Internal Names: When performing logic or saving data, always use the InternalName. The Title (Display Name) can be changed by users at any time, which will break your code if you rely on it.
  2. Handle List GUIDs: While getbytitle is easier to read, using the List GUID is more reliable. If a site owner renames a list, the GUID remains constant, preventing your API calls from failing.
  3. Pagination and Limits: If you have an exceptionally large number of columns (though rare), be aware of OData limits. However, for the Fields endpoint, the total count is usually small enough that this isn't an issue.
  4. Version Context: These REST patterns were established in SharePoint 2013. While they still work perfectly in SharePoint Online and SPFx development, always verify against the latest Microsoft Graph documentation if you are building modern, cross-platform applications.

Common Mistakes to Avoid

  • Filtering after the request: Avoid fetching all 200+ fields and then using JavaScript's $.grep or array.filter() to clean the data. This wastes bandwidth and processing power. Always filter at the source (the server) using $filter.
  • Ignoring Calculated Fields: If you use ReadOnlyField eq false, you will automatically exclude Calculated columns. If your application needs to display these, you will need to adjust your filter logic to include them while still excluding system IDs.
  • Case Sensitivity: Remember that OData filters in SharePoint REST are case-sensitive. hidden eq false may fail where Hidden eq false succeeds.

Frequently Asked Questions

How do I get only columns from a specific Content Type?

If your list uses multiple content types, you can query the content type specifically to get its fields: /_api/web/lists/getbytitle('List')/contenttypes('ID')/fields.

Can I filter fields by their data type via REST?

Yes! You can add and TypeAsString eq 'Text' to your $filter string to retrieve only single-line text columns.

Why does 'Created By' still show up in my results?

'Created By' (Author) and 'Modified By' (Editor) are often marked as ReadOnlyField eq true. If you want to include them but exclude other system fields, you may need to explicitly include them in your filter logic using an or statement.

Wrapping Up

Retrieving a clean list of columns via the SharePoint REST API doesn't have to be a trial-and-error process. By combining Hidden eq false and ReadOnlyField eq false with a strategic InternalName exclusion, you can create a streamlined data object that is perfect for your custom SharePoint solutions. For even more control, the SchemaXml filtering method provides a deep-dive approach to isolating truly custom, user-defined fields from the standard SharePoint schema.