Free Essay


In: Other Topics

Submitted By onnanapiando
Words 1298
Pages 6
Designing Parallel Programs

Automatic vs Manuel Parallelization

- Very often, manually developing parallel codes is a:
Time Consuming
Error-prone and
Iterative Process

- The most commonn type of tool used to automatically parallelize a serial program is a parallelizing compiler or pre-processor.

- A parallelizing compiler generally works in two different ways:

A. Fully Automatic
- The compiler analyzes the source code and identifies opportunities for parallelism.
- The analysis includes identifying inhibitors to parallelism and possibly a cost weighting on whether or not the parallelism would actually improve performance.
- Loops (do,for) are the most frequent target for automatic parallelization.

B. Programmer Directed
- Using "compiler directives" or possibly compiler flags, the programmer explicitly tells the compiler how to parallelize the code.
- May be able to be used in conjunction with some degree of automatic parallelization also.

- The most common compiler generated parallelization is done using on-node shared memory and threads (such as OpenMP).

Caveats (warning/caution) that apply to automatic parallelization:
1. Wrong results may be produced.
2. Performance may actually degrade.
3. Much less flexible than manual parallelization.
4. Limited to a subset (mostly loops) of code.
5. May actually not parallelize code if the compiler analysis suggests there are inhibitors or the code is too complex.

Understand the Problem and the Program

- The first step in developing parallel software: understand the problem that you wish to solve in parallel.
- Determine whether ot not the problem is one that can actually be parallelized
- Identify the program's hotspots
- Identify the program's bottlenecks
- Identify inhibitors to parallelism.
- Investigate other algorithms if possible.
- Take advantage of optimized third parallel party software and highly optimized math libraries available from leading vendors

- to break the problem into discrete "chunks" of work that can be distributed to multiple tasks (aka decomposition sad)

- 2 basic ways to partition computation work among parallel tasks:
1. Domain Decomposition
- the data associated with the problem is decomposed. Each parallel task then works on a portion of the data.

-1d; 2d (block, cyclic)

2. Functional Decomposition
- The focus is on the computation to be performed rather than the data manipulated by the computation.
- The problem is decomposed according to the work that must be done.
- Each task then performs a portion of the overall work.

- Ecosystem modeling
- Signal Processing
- Climate modeling

The need for communication between tasks depends upon your problem:

You DON'T need communications:
- problems that don't need for tasks to share data.
- composed of multiple tasks that can act independently of each other to do their portion of the work.
- these types of problems are often called embarrasingly parallel because they are so straight-forward. Very little intertask communication is required.

You DO need communications:
- most parallel application are not quite so simple, and do require tasks to share data with each other.

Factors to Consider:


1. Cost of communications:
- Inter-task communication virtually always implies overhead.
- Machine cycles and resources that could be used for computation are instead used to package and transmit data.
- Communications frequently requires some type of synchronization between tasks, which can result in spending time "waiting" rather than doing work
- Competiting communication traffic can saturate the network bandwidth, further aggravating performance problems

2. Latency vs Bandwidth:
- Latency: is the time it takes to send a minimal (0 bytes) message from point A to point B. it is usually expressed in microseconds.
- Bandwidth: is the amount of data that can be communicated per unit of time. usually expressed in gigabytes/sec or megabytes/sec
- sending many small messages can cause latency to dominate communication overhead.
- it is more efficient to send many small messages over large messages, thus increasing the effective communication bandwidth.

3. Visibility of communications:
- with the Message Passing Model, communications are explicit and generally quite visible and under control of the programmer
- with the Data Parallel Model, communications often occur tranparently to the programmer, particularly on the distributed memory architecture

4. Synchronous vs Asynchronous communications
- requires some type of "handshaking" between tasks that are sharing data. This can be explicitly structured in code by the programmer, or it may happen at a lower level unknown to the programmer.
- are often referred to as blocking communications since other work must wait until the communications have completed.

- allow tasks to tranfer data independently from one another.
- often referred to as non-blocking communication since other work can be done while the communications are taking place.
- greatest benefit: interleaving computation with communication

5. Scope of communications
A. Point-to-point
- involves two tasks; one being the producer/sender of data, and the other one is the receiver/consumer

B. Collective
- involves data sharing between more than two tasks, which are often specified as being members in a common group, or collective.
- broadcast, scatter, gather, reduction

6. Efficiency of communications
- communications performance
- Which implementation for a given model should be used?
- What type of communication operations should be used?
- Network media - some platforms may offer some more than one network for communications. Which one is the best?

7. Overhead and complexity


3 types of Synchronization
1. Barrier

2. Lock/Semaphore
3. Synchronous communication operations

Designing Parallel Programs

Data Dependencies

- a dependence exists between program statement when the order of program execution affects the result of the program

- a data dependence results from multiple use of same storage location/s in different tasks

- dependencies are important to parallel programming because they are one of the primary inhibitors to parallelism.

How to handle data dependencies

1. Distributed Memory Architecture
- communicate required data at synchrpnization point

2. Shared Memory Architecture
- synchronize read/write operations between tasks

Load Balancing
- refers to the practice of distributing approximate equal amounts of works among tasks so that tasks are kept busy all the time. Can also be considered as minimization of task idle time.

- it is important to parallel programming for the performance reasons. For example, if all tasks are subject to barrier sync point, then the slowest task will be the overall performance.

How to Achieve Load Balance
1. Equally partition the work each task receives
- array/matrix, evenly distribute the data set among the tasks.
- loops iterations, evenly distibute the iterations across the tasks.
- If a heterogenous mix of machines with varying performance characteristic are being used, be sure to use some type of performance analysis tool to detect any load imbalances.

2. Use dynamic work assignment
- certain classes results in load imbalances even if the data is evenly distributed among the tasks. * sparse arrays - some tasks will have actual data work while others have mostly zeros * adaptive grid methods - some tasks may need to refine their mesh while others don't. * n-body simulations - where some particles may migrate to/from their original task domain to another task; where paticles owned by some tasks requires more work than those owned by other tasks.

- scheduler-task approach
- algorithm which detects and handles load imbalances as they occur dynamically within the code.


Computation/Communication Ratio:

Fine-grain Parallelism
- R
- L
- F
- I
- I

Coarse-grain parallelism:
- Relatively large amounts
- H
- I
- H

Which is best?
- depends on the algorithm and the hardware environment in which it runs.


The Bad News:
- inhibitors
- requires order of magniude more time than memory operations
- may be immature or not available for all platforms
- write operations can result in overwriting
- read operations can be affected by the file server's ability to handle multiple read requests at the same time.
- can cause sever bottlenecks and even crash file servers

The Good News:
- parallel file systems are available: example: G

- MPI- 1996


debugging tools:
- T
- D
- I
- S

tools for parallel program performance analysis and tuning: (10)
- L
- T
- H
- O
- V
- V
- P
- m
- m
- m…...

Similar Documents

Free Essay

It 210 Week 7

...form: student (String), test1 (Integer), test2 (Integer), test3 (Integer) Analysis Processes   1. Display Input student name and test scores screen   2. Prompt for Student name   3. Prompt scores   4. Perform calculations   5. Provide student names, scores and total Input   1. Student Name   2. Test Scores Output   1. Display Student name, scores and total Declare StudentName as string Declare Test1 as integer Declare Test2 as integer Declare Test3 as integer Open “GRADES” For Output As TestScores Set Test1, Test2, Test3 = 0 Write “Enter students name and test scores.” Write “Enter ZZZ for students name when finished.” Input StudentName, Test1, Test2, Test3 While StudentName “ZZZ” Write TestScores, StudentName, Test1, Test2, Test3 Write “Enter students name and test scores.” Write “Enter ZZZ for students name when finished.” Input StudentName, Test1, Test2, Test3 End While Close TestScores 2. Display the contents of the file GRADES created in Problem 1. Each student’s record should appear on a separate line and include the total score (the sum of the three tests) for that student. For example, a line of output might be as follows: R. Abrams 76 84 82 242 Declare Sum as integer Open “GRADES” For Input As TestScores While Not (TestScores) Read TestScores, StudentName, Test1, Test2, Test3 Add Test1 + Test2 +......

Words: 286 - Pages: 2

Free Essay


...|Testing Access to Shares from Client Machines | |Overview |In this exercise you will attempt to access the shared folders created in exercise 9.3 from the | | |Windows 7 client machine using various accounts. | |Completion time |minutes | |Question 3 |Is PAckerman able to make changes to the Test1 document? | | | | | | | |Question 4 |Is Tyee able to make changes to the Test1 document? | | | | | | | 7. Take a screen shot of the Access Denied message for the Tyee account, by pressing Alt+ Prt......

Words: 627 - Pages: 3

Premium Essay


...In this homework, you will design a program to perform the following task: Criteria Exceeds Meets Does not meet Design 20 points (18-20 points) Employs Modularity (including proper use of parameters, use of local variables etc.) most of the time Employs correct & appropriate use of programming structures (loops, conditionals, classes etc.) most of the time Efficient algorithms used most of the time 17 points (15-17 points) Employs Modularity (including proper use of parameters, use of local variables etc.) some of the time Employs correct & appropriate use of programming structures (loops, conditionals, classes etc.) some of the time Efficient algorithms used some of the time 14 points (0-14 points) Rarely employs Modularity (including proper use of parameters, use of local variables etc.) Rarely employs correct & appropriate use of programming structures (loops, conditionals, classes etc.) Poorly structured and inefficient algorithms Functionality 40 points (36-40 points) Program fulfills all functionality All requirements were fulfilled Extra effort was apparent 35 points (29-35 points) Program fulfills most functionality Most requirements were fulfilled 28 points (0-28 points) Program does not fulfill functionality Few requirements were fulfilled Test 20 points (18-20 points) Comprehensive test plan 17......

Words: 282 - Pages: 2

Premium Essay


...TEXAS ROAD RULES TEST1 In a passenger car or truck (with a manufacturers rated carrying capacity of not more than 1500 pounds)which of the following must use safety belts?-only the driver and front seat passengers.2 If you have an accident in which someone is injured, you should?-carefully help anyone who might be hurt, and send for skilled help as quickly as possible.3 Vehicle skids are most likely to be caused by?-Driving too fast on slippery roads.4 A flashing red traffic light means?-Stop, look. And yield before proceeding.5 When you are taking medicine prescribed for you by a doctor you should?-Ask your doctor if it is safe for you to drive.6 If you move you should report your change of address within?-30 day.7 If you are driving at the speed limit and another car sounds his horn and starts to pass, you should?-Slow down and allow him to pass8 A solid yellow line on your side of the canter stripe means?-Do not pass.9 You must give a signal either by hand, arm, or by signal device?-Anytime you change lanes.10 When a vehicle ahead of you stops to let a pedestrian pass in front of it, you should?-Stay in line and wait until the vehicle ahead proceeds.11 When choosing your driving speed, the most important thing to consider is?-The condition of the weather, traffic, road, car and driver.12 The maximum fine for a first conviction of driving without a valid drivers license is?-$ 200.0013 A posted speed limit of 55 means-You may drive 55 mph only under favorable driving......

Words: 340 - Pages: 2

Free Essay

Letter Grade Flowchart

...Design a program to calculate the numerical and apathetical scores. 3. Test your program with the following data: Student: SJohnson Test1 95 Test2 88 Test 3 73 Solution Page [pic] Code: /** * This program will calculate the average score of three test for a specific student. * The output will be in both numerical and alphabetical form. * * @author Jeannie Ray) * @version (10/02/06) */ import java.util.Scanner; public class GradeCalculator { public static void main (String[] args) { // instance variables - int test1 = 0; int test2 = 0; int test3 = 0; int avgTest = 0; Scanner in = new Scanner (; //allow for user name input System.out.print ("What is your name? "); String input =; //allow user to input test scores System.out.print ("What is your score on the first test? "); test1 = in.nextInt(); System.out.println("Thank you!"); System.out.println(); System.out.print ("What is your score on the second test? "); test2 = in.nextInt(); System.out.println ("Thank you!"); System.out.println(); System.out.print ("What is your score on the third test?"); test3 = in.nextInt(); System.out.println ("Thank you!"); System.out.println(); //calculate the average score from all three tests avgTest = (test1 + test2 + test3)/3; if (AvgTest>89) System.out.println (input + " your numerical average equals " + avgTest + " which ......

Words: 383 - Pages: 2

Free Essay


... * Dayani Herrera * Week 2 Assignment The Chain of Custody Process Limiting Access to the Evidence * When a piece of evidence is collected, there are numerous individuals who will have access to it before the case goes to trial. The investigator, evidence technician,
property officer, laboratory technician, prosecutor and defense attorney all may have custody. Each individual in this chain has a responsibility for the safe-keeping and preservation of the evidence while it is in their possession.

 Initial Collection * Before any evidence is removed from the crime scene, there should be an evidence custodian designated as the person in charge of initiating and maintaining all evidence. This person needs to document the crime scene location, as well as his time of arrival. There should also be a crime scene log established, witness list and documentation of identification, arrival and departure times of all law enforcement crime scene personnel.

 Record Keeping * The investigator who handles the evidence initially should affix her name, badge number and assignment to the package containing the evidence. When the evidence leaves the officer's possession, a record should be made in her case notes identifying to whom the evidence was given, the date and time, and the reason it was turned over. A signed receipt should be obtained from the person accepting the evidence and another signed receipt should be obtained when the item is returned. When the......

Words: 667 - Pages: 3

Premium Essay


...1. Managerial accounting: A) is more future oriented than financial accounting. B) tends to summarize information more than financial accounting C) is primarily concerned with providing information to external users. D) is more concerned with precision than timeliness. 2. The function of management that compares planned results to actual results is known as: A) planning. B) directing and motivating. C) controlling. D) decision making. 3. Which of the functions of management involves overseeing day-to-day activities? A) Planning B) Directing and motivating C) Controlling D) Decision making 4. Ideally, how many units should be produced in a just-in-time manufacturing system? A) budgeted customer demand for the current week. B) budgeted customer demand for the following week. C) actual customer demand for the current week. D) maximum production capacity for the current week. 5. After careful planning, Jammu Manufacturing Corporation has decided to switch to a just-in-time inventory system. At the beginning of this switch, Jammu has 30 units of product in inventory. Jammu has 2,000 labor hours available in the first month of this switch. These hours could produce 500 units of product. Customer demand for this first month is 400 units. If just-in-time principles are correctly followed, how many units should Jammu plan to produce in the first month of the switch? A) 370 B) 400 C) 430 D) 470 6. Management accounting focuses...

Words: 1225 - Pages: 5

Free Essay

Nmq8 Revision Question

...describe various types of partitions in Linux.[8] 4. [10] i. With the aid of drawings, describe the Linux file system structure. Identify some directories that make the structure.[5] ii. State and discus two types of Linux file system.[5] 5. Discuss steps of the Linux booting process.[10] 6. [10] a. What does the following command do? [2] $ chmod u+x book.tb b. Given the file permission “-rwx--xr--” [2] (i) What is the file type? (ii) Write the file permission in absolute mode. c. What is a standard input file [1] d. What is a standard output file [1] e. What do the following commands do? [4] (i) $ cat 0 < test1 [1] (ii) $ cat test1 > test2[1] (iii) $ ls | sort –r |cat test1 [2] 7. Write a bash script that will display the following output [10] 1-5-9-13-17 ...

Words: 431 - Pages: 2

Premium Essay


...thousands of substances and products. The group’s products are used extensively by customers in the chemical, environmental, pharmaceutical, biotech, academic, and government sectors. 5 Measures of Success No other company offers the breadth and depth of measurement tools and expertise to meet the world’s critical requirements for electronic and bio-analytical measurement. With our impressive track record and exclusive focus on measurement, Agilent is the global industry leader — by virtually any measure. Agilent’s electronic measurement business is the leader in numerous product markets: #1 in •test instrument sales1 #1 in •communications test1 Measurement solutions for fiber optic, broadband, radio frequency, and microwave communications networks and products: #1 in •cellular mobile test1 #1 in •digital cellular mobile test1 #1 worldwide • in wireless communications test equipment1 #1 worldwide • in wireless communications one-box test sets for manufacturing2 #1 worldwide • in wireless mobile manufacturing test equipment2 #1 worldwide • supplier for custom and tailored systems in the wireless market #1 worldwide • supplier for manufacturing test diagnostic tools for the communications market #1 worldwide • in signal routing and switching products1 #1 worldwide • in optical test equipment2 #1 worldwide • in network analyzers1 #1 worldwide • in metro network test equipment3 #1 in •ATM/IP/router transmission test #1 in •pulse/data generators1 High-frequency......

Words: 2239 - Pages: 9

Free Essay


...You should answer all of the questions listed below as they relate to the financial statements of any U.S. public company of your choice in its latest annual report. You can find the financial statements of any U.S. public company by visiting Morningstar, Yahoo Finance or MSN Money, and using the stock market symbol for your chosen company (which can be looked up on Morningstar, Yahoo Finance or MSN Money). If you work for a U.S. public company, you may want to use the financial statements of your company - but the choice is up to you. It is useful to compare financial ratios for a company with financial ratios of its industry. Industry financial ratios can be found on, Yahoo Finance and MSN Money. 1) What 3 items of important information does the income statement reveal about the financial performance of the company over the last three years? The income statement shows that HCA is making a profit. Example below Revenue = 36,918,000 Expenses = 25,091,000 profit = 11,827,000 Income statement can also help spot trends and turnarounds when compared to previous periods. It shows that the company has been making a profit the last two years 2) What 3 items of important information does the balance sheet reveal about the financial position of the company over the last two years? Answer shown in excel below 3) Can you identify the major sources of funding used by the company from the information presented in the company's annual report? If not,......

Words: 595 - Pages: 3

Free Essay


...Four types of mandatory parameters: 1. Value: declared without any modifiers 2. Ref: ref modifier 3. Out: out modifier 4. Param: declared with the params modifier REF&OUT: Advantage: method can change multiple variables Disadvantage: they allow multiple methods to have access to the same data Parameter array: Local array declared within the method header by using the keyword params. Only one param keyboard is permitted in a method declaration Week 3 Overloading: Involves using one term to indicate diverse meanings When you overload a c# method, you can write multiple methods with a shared name. The compiler understands your meaning based on the arguments you use with the methods Methods are overloaded correctly when they have the same identifier but difference parameter lists Overloaded only means that you have more than one method signature Ex: DisplayWithBorder (int) and DisplayWithBorder (string) Overload resolution: determines which method to execute when a method call could execute multiple overloaded methods Applicable methods: a set of methods that can accept a call with a specific list of arguments Betterness rules: that determine which method version to call Ambiguous methods: a situation which the compiler cannot tell which method to use. Occurs when you overload a method ex simplemethod (inum, inum); // error! Call is ambiguous Optional parameter: only value params can be given default values. All optional parameters in a list...

Words: 301 - Pages: 2

Premium Essay


...My thoughts about Grandma Mary, the epitome of love. I want to say to you all the things I wish I’d said to my other loved ones who left too soon, before I could make sure to tell them all they meant to me. Grandma Mary, you are one of a kind. Your life is a reminder of the best attributes of the people who built our great country, an example for those who will come after you. You’re a reminder of the virtues that matter, a precious treasure to all who know you. Though you’d be quick to remind me you’re not perfect and have your human faults, it could not diminish all the wonderful things you are. Godly Beautiful Thankful Wise Tender Loving Giving Frugal Fun Good Precious Caring Kind Dignified Polite Unselfish Tough Laughter Commitment Honest Devotion Independent Faithful I wrote this when we were visiting you and I showed it to Tina, I was just trying to list all the fruits of the spirit that you have so consistently demonstrated with your beautiful life. Now I realized I never shared it with you, so here it is… I love you, Grandma. I never really had a Grandma growing up, as my my Mom’s mother died when I was young, and my father’s mother had dementia by the time I was a teenager, and even the childhood years only hold a few memories of her. I was the middle child of eight children, so there was a lot of competition to get attention from adults, and I never imagined having a Grandma like you, who always has a hug and a kiss, along with......

Words: 317 - Pages: 2

Free Essay


...Test  2     Math  510  Fall  2015   All  work  must  be  competed  on  your  own.     19  Nov  2015     Complete  the  following  work  using  R  Studio  and  submit  the  rmd  file  to  Github.   Name  the  file  TEST2_R.  Make  sure  to  include  comments.     Using  the  abalone  data:     1.   Take  samples  of  1%,  10%,  25%,  50%,  80%,  and  100%  of  the  data.  Calculate   the  min,  max,  mean  and  variance  of  the  Whole_wgt  column  from  each  sample   and  store  in  a  dataframe.  There  should  be  6  columns  and  four  rows  in  the   dataframe  (a  column  for  each  sample  size  and  a  row  for  each  measure).   Name  the  mean  row  SampleMean,  the  other  rows  appropriately,  and  the   columns  based  on  the  %  sample.     2.   Create  a  grid  of  histograms  that  contains  all  sample  sizes.     3.   Implement  a  loop  that  takes  1,000  samples  at  each  of  the  sample  sizes  in  1   above  (except  the  100%  sample!).  Calculate  a  mean ......

Words: 330 - Pages: 2

Premium Essay


...1. How has Islam shaped conceptions of music for the peoples of the Middle East? In the Middle East, Islam is not just a religion; it is a way of life. Islam’s influence spread in the Middle East music conceptions. For example, the Arabic scale is strongly melodic, based around various maqamat or modes. Also, Mann explains that “under the wide rule of Islam, vulgar lyrics and suggestive dancing by women became illegal. Much post-Islamic music is used in ceremonial dance and recreation. (Mann)” In the Middle East, people may hear their songs of praise and prayer. The “World Music: A global Journey, says that “in addition to being suspicious of the sensual aspects of music Islamic aesthetics frowns on realistic representation in art, especially of the human form, to avoid the temptation toward idol worship. But as with sound, it may not always be obvious to outsiders what is considered art and what is not. (Miller 255)” Because, the influence of Islam has had great impact on people related with music, and people used to meditation, trance, and self-flagellation with one’s a higher sense of God while listening music. 2. What is modal improvisation? Is it primarily a compositional or a freely expressive form of performance? According to Henry, modal improvisation is that “a typical modal tune may have only two or three chords, and each may last 8 or even 16 measures. In one sense, modal playing is much easier than playing changes, since it does not require your brain to do......

Words: 295 - Pages: 2

Premium Essay


...1. What is the supreme law of the land? A: The Constitution 2. What does the Constitution do? A: sets up the government A: defines the government A: protects basic rights of Americans 3. The idea of self-government is in the first three words of the Constitution. What are these words? A: We the People 4. What is an amendment? A: a change (to the Constitution) A: an addition (to the Constitution) 5. What do we call the first ten amendments to the Constitution? A: The Bill of Rights 6. What is one right or freedom from the First Amendment?* A: speech A: religion A: assembly A: press A: petition the government 7. How many amendments does the Constitution have? A: twenty-seven (27) 8. What did the Declaration of Independence do? A: announced our independence (from Great Britain) A: declared our independence (from Great Britain) A: said that the United States is free (from Great Britain) 9. What are two rights in the Declaration of Independence? A: life A: liberty A: pursuit of happiness 10. What is freedom of religion? A: You can practice any religion, or not practice a religion. 11. What is the economic system in the United States?* A: capitalist economy A: market economy 12. What is the "rule of law"? A: Everyone must follow the law. A: Leaders must obey the law. A: Government must obey the law. A: No one is above the law. B. System of Government 13. Name one branch or part of the government.* A: Congress A: legislative A:......

Words: 2497 - Pages: 10