That allowed me to better handle the file(s) that fail without stopping the rest of the data imports. I decided to loop with a macro processing them one at a time. That meant I needed to rerun the step from scratch, which can take a while if pulling text files from a far. I found that when dealing with hundreds of files, I was almost certainly getting one or two that were irregularly formed and caused my data step to stop. On reading external files with the wildcard notation, I found this is great for text files with a stable structure.
HOW TO SEARCH MULTIPLE FILES FOR TEXT DRIVER
I thought PANELBY orders the values in the filename column, making it drivehome, kitchentable, lawncut, runhome, trampoline? Although, that makes you a very smooth driver but erratic mower! We'll have fewer blocks of repetitive code, and we can accomplish more across all of these cases before we have to resort to SAS macro logic to repeat operations for each file.Īs a simple example, I can create a simple visualization with a single PROC SGPANEL step.
HOW TO SEARCH MULTIPLE FILES FOR TEXT CODE
SAS procedures support CLASS and BY statements that allow us to simplify our code when reporting across different groups of data. In this single data set, we still have the information that keeps the records distinct (the name of the original files), so we haven't lost anything. (But if you must split a single data set into many, here's a method to do it.) While there might be good reasons to do that for some types of data, I believe that we have more flexibility when we keep all of these records together in a single data set. Managing data files: fewer files is betterīecause we started this task with 5 distinct input files, it might be tempting to store the records in separate tables: one for each accelerometer case. In the output, you'll notice that we now have the fully qualified file name that SAS processed using INFILE. * store the name of the current infile */ infile tsvs filename=tsvfile Length casefile $ 100 /* to write to data set */ Here's my code, refactored a little bit, now with the options for reading/storing the file names we're processing:įilename tsvs "/home/chris.hemedinger/tsv/*.tsv" So, you need to include code that assigns the value to a permanent variable that you can save.
Like all automatic variables in the DATA step, this variable won't be written to the output data set by default. The FILENAME= option allows you to specify an automatic variable to store the name of the current input file. The DATA step INFILE statement supports a special option for this - it's the FILENAME= option. If your input data files don't contain an eponymous field name, then you will need to use a different method to keep track of which records come from which files. A quick PROC FREQ shows the allocation of records for each case that I collected. It's convenient that my accelerometer app also captured the name of each TSV file so that I can keep these cases distinct. With a single data set that has all of my accelerometer readings, I can easily segment these with a WHERE clause in later processing. The SAS log shows which files have been processed and added into my data set. Infile "/home/chris.hemedinger/tsv/*.tsv" I also changed the name of the data set from "drive" to the more generic "accel". I replaced the "drivehome.tsv" filename with *.tsv, which tells SAS to match on all of the TSV files in the folder and process each of them in turn. I can read all of my TSV files in a single step by using the wildcard notation in my INFILE statement. And if I later come back and add more files to my TSV collection, I'll need to copy-and-paste the same code blocks for my additional cases. If I want to add any additional logic into my DATA step, that change would need to be applied 5 times. But as the number of code lines grows, so does the maintenance work. After all, copy-and-paste is a tried and true method for writing large volumes of code.
To import each of these data files into SAS, I could simply copy and paste my code 4 times and then replace the name of the file for each case that I collected. Here's my file directory in my SAS OnDemand for Academics account:
In addition to my commute, I collected data about 4 other activities, and thus accumulated a collection of TSV files. In my research, I didn't stop with just my drive home. Infile "/home/chris.hemedinger/tsv/drivehome.tsv"
0 kommentar(er)
