For those that haven't used SharePoint Document Sets before, here's a good introduction to it : Introduction to Document Sets
In Nintex Workflow, to query for documents in a document set, you use a Query List action.
I've found the easiest way to do this, is to add a Query List action to your workflow, configure it by selecting a list that contains your document set then add a field that you want to extract (usually the ID field). Finally, check the box to go into CAML Editor mode.
This is where you'll want to make some modifications to the CAML query. First of all, you need to know your document set name that you want to query. You don't need to know it as design time. This value can come in from a workflow variable. This is what you will be filtering on, as we want to find all the documents within the document set, the filter will be based on the ServerUrl of each document and to make sure it contains the following /[docsetname]/. Notice the forward slashes.
One final thing to note, is that document sets are very similar to folders in a library. Therefore, in order to have a CAML query look further than just the root of the library, you need to change the scope and tell it to look recursively.
In my test, I had a document set named DocSet, which resulted in the following CAML query.
You can test this out by putting that into a Query List action and clicking on the Run Now button to make sure you get the results you need.
There is a possible issue with this query. If you multiple document set, eg DocSet, DocSetA and DocSetB, and you have a folder or a document set within a document set with the same name, it will find those also.
The workaround, would be to have a longer ServerUrl that you compare to. eg /[listname]/[docsetname]/.