Have you seen anytime data inside the parquet? Very imp if we want to view the content we can install 'parquet-tools' brew install parquet-tools parquet-tools head filename We can always read the parquet file to a data frame in Spark and see the content. They are of columnar formats and are more suitable for analytical environments, write once and read many. Parquet files are more suitable for Read intensive applications. what is dataflow and why we are using it? Very imp Transformations on data. data-flow is a path for data to move from one part of the information system to another. Tell me end to end steps which you followed for copy data activity? Very imp Lookup-->For Each🡪copy activity🡪stored procedure (to log the audit details into table). What is getmetadata?
Get metadata is nothing but data about data. We can use this to check the header level details and it works on blob, sql db. and the files.
what is lookup and the available options? Very imp Lookup activity reads and returns the content of a configuration file or table. It also returns the result of executing a query or stored procedure. The output from Lookup activity can be used in a subsequent copy or transformation activity if it's a singleton value. The output can be used in a ForEach activity if it's an array of attributes. The Lookup activity can return up to 5000 rows; if the result set contains more records, the first 5000 rows will be returned
Can we use iteration inside the iteration? Very imp NO
What is delete activity and have you used in your project? Very imp After processing my files from my input folders we used to clean up the processed files but before that we will move into archive and next delete activity configures. It will do the processed files deletion from specified path
